Frequently Asked Questions and Troubleshooting
While following through our tutorials, you might have had questions regarding the details of the requirements and specifications. We have listed out the frequently asked questions and as a separate document. Please refer to the relevant information below:
Format Requirements
General requirements for a model
- A model should be in safetensors format.
- The model should NOT be nested inside another directory.
- Including other arbitrary files (that are not in the list) is totally fine. However, those files will not be downloaded nor used.
| Required | Filename | Description |
|---|---|---|
| Yes | safetensors | Model weight, e.g. model-00001-of-00002.safetensors |
| Yes | config.json | Model config file that includes the architecture. (Supported Models on Friendli) |
| Yes | tokenizer.json | Tokenizer for the model |
| No | tokenizer_config.json | Tokenizer config. This should be present & have a chat_template field for the Friendli Engine to provide chat APIs |
| No | special_tokens_map.json |
3rd-party account integration
How to integrate a Hugging Face account
- Log in to Hugging Face, then navigate to user settings → access tokens → User Access Tokens. Acquire a token.
- You may use a fine-grained token. In this case, please make sure the token has view permission for the repository you’d like to use.
- Integrate the key in Friendli Suite → User Settings → Account → Integrations
note
- If you revoke / invalidate the key, you will have to update the key in order to not disrupt ongoing deployments, or to launch a new inference deployment / fine tuning job
How to integrate a W&B account
- Log in to W&B, then navigate to user settings → danger zone → API keys. Acquire a token.
- Integrate the key in Friendli Suite → User Settings → Account → Integrations
note
- If you revoke / invalidate the key, you will have to update the key in order to not disrupt ongoing deployments, or to launch a new inference deployment / fine tuning job
Extra: How to upload a safetensors format model to W&B using W&B CLI
- Install the cli and log in using the API key → Command Line Interface | Weights & Biases Documentation
- Upload the model as an W&B artifact using the command below
wandb artifact put -n project/artifact_id --type model /path/to/dir
- With all this, the W&B artifact will look like this:
Using 3rd-party model/datasets
How to use a W&B artifact as a model
- Use the full name of the artifact
- The artifact name must be in the format of:
org/project/artifact_id:version
How to use a Hugging Face repository as a model
- Use the repository id of the model. You may select the entry from the list of autocompleted model repositories.
- You may choose specific branch, or manually enter a commit hash.
Troubleshooting
Can’t access the artifact
- The artifact might be nonexistent, or hidden so that you cannot access it.
You don’t have access to this gated model
- The repository is gated. Please follow the steps and gain approval from the owner using Hugging Face Hub.
The repository / artifact is invalid
- The model does not meet the requirements. Please check if the model follows a correct safetensors format.
The architecture is not supported
- The model architecture is not supported. Please refer to Supported Models on Friendli.
- In case of a fine-tuning job, you’re choices are limited to Llama and Mistral models, under the 80GB size limit.