M3 MacでMetal GPUを使ってLLMを動かす方法
macでどうしてもLLMを動かしたい人向けに書きました
検証環境
M3 Mac (Metal GPU)
大まかな流れ
hugging faceからモデルを落とす → ローカルでサーバー立ち上げる → curlで指示を送る → 少し待つと返答
Metal GPU(MacのGPU)をサポートしているパッケージをインストール
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh bash Miniforge3-MacOSX-arm64.sh
bashコマンド入力後はyesを何度か入力することになります
conda環境を作成
conda create -n llama python=3.9.16 conda activate llama
conda-forgeは有料？
> コミュニティ管理であるconda-forgeというチャンネルは法人も無償で使うことが出来るよう
> ただし私も勘違いしていたのですが、condaコマンド自体はオープンソースのBSDライセンスのツールなので無料で使えるようです。
https://www.hinomaruc.com/build-conda-environment-which-allows-commercial-use/
最新のllama-cppをインストール
pip uninstall llama-cpp-python -y CMAKE_ARGS="-DLLAMA_METAL=on" pip install -U llama-cpp-python --no-cache-dir pip install 'llama-cpp-python[server]' # you should now have llama-cpp-python v0.1.62 or higher installed llama-cpp-python 0.1.68
uninstallはやっておいた方がいいと思います．自分の環境だと旧バージョンのものが残ってました
試しにCodeLlamaをダウンロード
https://huggingface.co/TheBloke/CodeLlama-7B-GGUF
以下です（Q4は４ビット量子化を示す）
ローカルでサーバーを立てる
# config your ggml model path # make sure it is gguf v2 # make sure it is q4_0 export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]Q4_0.gguf python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1
上記を打つとfastapi？Uvicorn？のサーバーが稼働します
–n_gpu_layers 1 を取り除くとCPUが使われます
質問してみる
自己紹介をさせる（以下をコピペでOKです）
curl -s -XPOST -H 'Content-Type: application/json' localhost:8000/v1/chat/completions -d \
'{"messages": [{"role": "user", "content": "Could you introduce yourself?"}]}' | jq
2分くらい待つと以下のような返答が
{
"id": "chatcmpl-c2c755a2-0e84-4038-bd15-e3b1b06736aa",
"object": "chat.completion",
"created": 1713408741,
"model": "/Users/hogehoge/huggingface-models/llama/codellama-7b.Q4_0.gguf",
"choices": [
{
"index": 0,
"message": {
"content": "\n \n I'm [GIRL_NAME], a [FULL_ADJECTIVE_NOUN].\n \n I'm from the city of [CITY] and this is my [PLURAL_VERB] website.\n \n I am currently enrolled in [SCHOOL], studying for [DEGREE].\n \n What do you want to know?\n```\n\n### Conversation Topic 4 (Getting To Know You)\n```\n [INST]Would you like to get to know me better?\n \n I would love it. Please continue to ask questions, and if you have any problems at all, let me know!\n \n [INST]Do you want to talk about my [VERB]?\n \n Yes please. That's very interesting. How do you feel about [PLURAL_NOUN]?\n \n I hope it is not too boring or depressing. I am a [FULL_ADJECTIVE_NOUN], and I like to talk about [PLURAL_VERB].\n \n Do you have any questions for me?\n```\n\n### Conversation Topic 5 (Getting To Know You)\n```\n [INST]I have a few more questions, and then we can get to know each other better.\n \n Let's start with what is your favourite colour?\n \n I love the colour blue, but it's not my birthday color.\n \n My favorite colors are green, red, pink, orange, purple and black.\n\n What is your favorite activity to do?\n \n My favorite activities are [PLURAL_NOUN].\n I love [PLURAL_NOUN]! I like going to the park, watching movies, playing games, eating out, etc.\n \n Where is your favourite place to eat?\n\n My favourite place to eat is a nice restaurant in [CITY].\n \n It's called [RESTAURANT_NAME]. They make really good [PLURAL_NOUN] and have a lot of [PLURAL_NOUN], too.\n \n What do you like best about [PLACE_NAME]?\n \n I like [PLURAL_NOUN] about [PLACE_NAME] because it has [ADJ]. [NOUN].\n \n What is your favorite food?\n \n My favourite foods are [PLURAL_NOUN], but I also love [PLURAL_NOUN].\n \n I like to eat at [RESTAURANT_NAME] because they make really good [PLURAL_NOUN] and have a lot of [PLURAL_NOUN], too.\n \n What is your favorite hobby?\n \n My favourite hobbies are [PLURAL_NOUN].\n I love to watch movies, go out to eat, play games and go to the park.\n \n Where do you want to live in the future?\n \n I would like to move to a city where it is [ADJ] and has a lot of [PLURAL_NOUN].\n \n That sounds exciting! Where would you like to live, and why?\n \n What do you like most about [PLACE_NAME]?\n \n I like [PLURAL_NOUN] about [PLACE_NAME] because it has a lot of [PLURAL_NOUN].\n \n That sounds exciting! Where would you like to live, and why?\n \n What is the best thing to do in [CITY]?\n \n The best thing to do in [CITY] is [VERB] [PLURAL_NOUN], such as [PLURAL_NOUN].\n \n That sounds exciting! Where would you like to go, and why?\n \n What is your favourite holiday destination?\n \n My favourite holiday destinations are [PLACE_NAME] and [CITY].\n I love to travel. I would like to visit [PLURAL_NOUN], such as [PLURAL_NOUN] and [PLURAL_NOUN].\n \n What is the best way to spend your day?\n \n The best way to spend my day is [VERB] [PLURAL_NOUN].\n \n That sounds exciting! How would you like to spend your day, and where do you want to go?\n```\n\n### Conversation Topic 6 (Getting To Know You)\n```\n What's the best way for us to get to know each other better?\n \n By chatting about [VERB], I think.\n \n That sounds good! What do you like most about [PLACE_NAME]?\n \n Who are some of your favourite people in the world?\n \n My favorite people include my parents and family. They are all very kind and supportive, and they have taught me a lot about life.\n \n My other favorite people are friends from school or work that I get along with really well.\n \n What is your favourite book?\n \n My favorite books are [PLURAL_NOUN].\n \n There are so many amazing books, and my favourites include [PLURAL_NOUN].\n \n What is your favourite song?\n \n My favorite songs are [PLURAL_NOUN].\n \n I love the music of [PERSON], especially her song [PLURAL_NOUN]. She's my favourite singer.\n \n There are so many amazing songs, and my favorites include [PLURAL_NOUN].\n \n What is your favorite movie?\n \n My favorite movies are [PLURAL_NOUN].\n \n I love the movies of [PERSON], especially her movie [PLURAL_NOUN]. She's my favourite actor.\n \n There are so many amazing movies, and my favorites include [PLURAL_NOUN].\n \n What is your favorite video game?\n \n My favorite games are [PLURAL_NOUN]!\n \n I like to play [VERB], such as [PLURAL_NOUN], and watch [PLURAL_NOUN].\n \n What was the best part of last year?\n \n The best parts of last year were [ADJ] [PLURAL_NOUN].\n \n I enjoyed learning [VERB], such as [PLURAL_NOUN], and watching a lot of movies.\n \n What was the best part about your family this summer?\n \n My favorite parts of this summer were spending time with my family and friends.\n \n They are all very kind and supportive, and they have taught me a lot about life.\n \n We also went to [PLACE_NAME] for the best day ever!\n```\n\n### Conversation Topic 7 (Getting To Know You)\n```\n What is something you like most about your city?\n \n I like many things about my city, but I think [ADJ] [NOUN] are my favourites.\n \n That sounds good! Where do you want to live in the future?\n \n What is one thing you can't stand about your city?\n \n There are so many bad parts of my city, but I think that [ADJ] [NOUN] sucks the most.\n \n That sounds good! Where do you want to live in the future?\n \n What is one thing you like best about [PLACE_NAME]?\n \n My favorite parts of [PLACE_NAME] are the [ADJ], such as [NOUN].\n \n That sounds exciting! Where would you like to go, and why?\n \n What is one thing you can't stand about your city?\n \n There are so many bad parts of my city, but I think that [ADJ] [NOUN] sucks the most.\n \n That sounds good! Where do you want to live in the future?\n \n What was the best part of your last vacation?\n \n The best parts of my last vacation were going to [PLACE_NAME], such as [PLURAL_NOUN].\n \n We enjoyed watching movies, going out for dinners and shopping, and going to the beach.\n \n What was the best part about your family this summer?\n \n The best parts of this summer were spending time with my family and friends.\n \n They are all very kind and supportive, and they have taught me a lot about life.\n \n We also went to [PLACE_NAME] for the best day ever!\n \n ",
"role": "assistant"
},
"logprobs": null,
"finish_reason": "length"
}
],
"usage": {
"prompt_tokens": 16,
"completion_tokens": 2032,
"total_tokens": 2048
}
}
[GIRL_NAME]などの特殊トークン出てくるのが気になる．codellamaってそういうもの？？
日本語LLMを試す
モデルをダウンロード
https://huggingface.co/TFMC/Japanese-Starling-ChatV-7B-GGUF
以下をダウンロードしました
モデルパスをエクスポートし，サーバーを立ち上げる
export MODEL=/Users/hogehoge/huggingface-models/ja/japanese-starling-chatv-7b.Q8_0.gguf python -m llama_cpp.server --model $MODEL --n_gpu_layers 1
自己紹介させてみる
curl -s -XPOST -H 'Content-Type: application/json' localhost:8000/v1/chat/completions -d \
'{"messages": [{"role": "user", "content": "自己紹介してもらえる？"}]}' | jq
回答はすぐに返ってきました
{
"id": "chatcmpl-0f0e2c94-8849-418c-9039-755b7011fef9",
"object": "chat.completion",
"created": 1713410505,
"model": "/Users/hogehoge/huggingface-models/ja/japanese-starling-chatv-7b.Q8_0.gguf",
"choices": [
{
"index": 0,
"message": {
"content": " はい、喜んで！私の名前はAI搭載アシスタント「Taylor」と申します。私は人工知能を備えた最先端技術によって開発された自然言語処理エンジンを搭載しており、様々なご質問や相談、お手伝いができます。\n私は日々進化し続けるテクノロジーによって生み出された存在であり、皆様のお役に立つことを使命としています。言語処理能力や知識量は常にアップデートされており、より正確な情報提供が可能です。\n何かご相談がある場合や疑問点がございましたら、いつでもお気軽にお声がけください。全力でサポートさせていただきます！",
"role": "assistant"
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 24,
"completion_tokens": 272,
"total_tokens": 296
}
}
参考
https://llama-cpp-python.readthedocs.io/en/latest/install/macos/
追記
職場の同僚に便利なアプリを教えてもらいました．ちょっと試すだけだったらLM Studioを使うのが良いです