Allow huggingface_hub 2.0

[musicinmybrain]

I didn’t review changelogs or anything; I just ran make test in binders/python and observed that everything still passes.

[chrikrah]

@musicinmybrain The cap decides what pip install tokenizers pulls in, and 2.x moves two things under it. The HTTP stack goes from httpx to httpx2. The huggingface-cli console script goes away. Neither one reaches bindings/python/src/tokenizer.rs:158, the only runtime call site in the binding.

Wauplin cleared the previous bump in #1866 on the three keywords he named there. #2391 took that call to ten, and all ten still resolve on released 2.0.0 and 2.1.1. The cache layout and the FileNotFoundError behind tests/test_from_pretrained.py:68 and :83 hold too. Nothing in bindings/python blocks <3.0.

# tokenizers at b724e8b. 1.33.0 is what today's `<2.0` allows, 2.1.1 the top of `<3.0`.
$ pip download --no-deps huggingface_hub==1.33.0 huggingface_hub==2.1.1
$ unzip -p huggingface_hub-1.33.0-py3-none-any.whl '*/METADATA' | grep -E '^Requires-Dist: (httpx<|httpx2)'
Requires-Dist: httpx<1,>=0.23.0
$ unzip -p huggingface_hub-2.1.1-py3-none-any.whl '*/METADATA' | grep -E '^Requires-Dist: (httpx<|httpx2)'
Requires-Dist: httpx2<3,>=2.0.0
$ unzip -p huggingface_hub-1.33.0-py3-none-any.whl '*/entry_points.txt' | sed -n '2,4p'
hf = huggingface_hub.cli.hf:main
huggingface-cli = huggingface_hub.cli.deprecated_cli:main
tiny-agents = huggingface_hub.inference._mcp.cli:app
$ unzip -p huggingface_hub-2.1.1-py3-none-any.whl '*/entry_points.txt' | sed -n '2,4p'
hf = huggingface_hub.cli.hf:main
tiny-agents = huggingface_hub.inference._mcp.cli:app

$ python3 -m venv v && v/bin/pip install huggingface_hub==2.1.1
$ v/bin/python - <<'PY'
import inspect, pathlib, tempfile, huggingface_hub as h
kw = dict(repo_id="some-org/some-model", filename="tokenizer.json", revision="main", token=None,
          cache_dir=None, force_download=False, local_files_only=True, subfolder=None,
          library_name="tokenizers", library_version="0.23.0")   # tokenizer.rs:148-157
d = pathlib.Path(tempfile.mkdtemp()); c = "0" * 40
r = d / "models--some-org--some-model"                           # test_from_pretrained.py:68
(r / "refs").mkdir(parents=True); (r / "refs" / "main").write_text(c)
(r / "snapshots" / c).mkdir(parents=True); (r / "snapshots" / c / "tokenizer.json").write_text("{}")
kw["cache_dir"] = d
print(h.__version__, "| missing kwargs:", [k for k in kw if k not in inspect.signature(h.hf_hub_download).parameters])
print("cache hit ->", type(h.hf_hub_download(**kw)).__name__)
kw["repo_id"] = "some-org/does-not-exist"                        # test_from_pretrained.py:83
try:
    h.hf_hub_download(**kw)
except Exception as e:
    print("missing repo ->", type(e).__name__, "| FileNotFoundError:", isinstance(e, FileNotFoundError))
PY
2.1.1 | missing kwargs: []
cache hit -> str
missing repo -> LocalEntryNotFoundError | FileNotFoundError: True
# identical output on 2.0.0. CPython 3.12.3.
# not run: the binding itself, which I did not build

non-blocking: bindings/python/examples/train_parity_bpe.py:14 tells the reader to run huggingface-cli login, a command a 2.x install no longer ships. tokenizer.rs:128 already documents hf auth login.

@ArthurZucker, you merged #1385 and #1866 on this same line, both bumping the cap and keeping it. Does <3.0 get the same call, or do you want it held until hub 2.x settles?

[SBrandeis]

This should be safe, we only use hf_hub_download and its behavior is stable between the two majors
Tested locally as well

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hanouticelina]

this cap currently breaks our huggingface_hub 2.x prerelease CI in transformers (dependency resolution fails, eg: here). Since transformers pins tokenizers<0.24.0, would it be possible to cut a 0.23.3 with just this pin change? 🙏

cc @SBrandeis @ArthurZucker @McPatate

[ArthurZucker]

Yep that's planned