pip install private package 403 azure databricks

I'm trying to install a package from a private jfrog artifactory repo using the following command

pip install -i https://user:pass@com.jfrog.io/artifactory/api/pypi/private-pypi/simple --trusted-host com.jfrog.io private_package==0.1.1

and getting a 403 when running it within Azure Databricks. Its using Python 3.9 and pip 21.2.4

The exact same command works from another environment in AWS Databricks using Python 3.10 and pip 22.x. It also works from my local in Python 3.10 and pip 23.x. I created a docker container with the exact same version of pip (21.2.4) and python (3.9) as the Azure Databricks environment and the installation works.

In the Azure Databricks environment is also set

pip config set global.index-url https://user:pass@com.jfrog.io/artifactory/api/pypi/private-pypi/simplepip config set global.trusted-host com.jfrog.io

with no success

I've also checked the values for HTTPS_PROXY, https_proxy, HTTP_PROXY, and http_proxy and they are all empty.

I also tried installing a different package from our private artifactory in the Azure Databricks environment (using different credentials) and that works.

I'm not sure what could be happening here, can anyone help?