PHakosikristo
Established
How to install AI (Large Language Model) and host it to your PC offline.
For educational purpose only, this is for you to host an AI (LLM) in your own machine.
Minimum System Requirement
RAM - 8GB (16GB recommended)
VRAM - Any VRAM pwede ding wala pero much better if meron since dito nangyayari yung computing and hindi sa processor. Your GPU will handle all the lifting much better if NVIDIA card gamit mo, the better the GPU the faster the response.
Understanding the AI Model and quantization.
Example :
FP16 (Original 16-bit): Crisp, pristine quality, but the file is massive (e.g., ~14 GB for a 7B model).
8-bit (Q8_0): Virtually identical quality to FP16, but cuts memory usage in half (~7.5 GB)
4-bit (Q4_K_M): 1080p equivalent—slight drop in fine detail, but uses 70% less VRAM (~4.5 GB).
2-bit (Q2_K): 480p equivalent—blurry, highly compressed, and prone to losing logic or factual details.
Lets say "Gemma 4 26B A4B QAT" yung napili mong model.
26B - Means it has 26 billion parameters. Parang ito yung overall brain capacity nung AI model.
A4B - Instead of puling 26 billion parameters (which is super bigat) the model smartly pulls 4 billion parameters only per token. Bali yung have the brain of 26 billion but with a speed of 4 billion model.
QAT - Medyo malalim na to eh, pero in short word eto yung training model na ginamit.
For educational purpose only, this is for you to host an AI (LLM) in your own machine.
Minimum System Requirement
RAM - 8GB (16GB recommended)
VRAM - Any VRAM pwede ding wala pero much better if meron since dito nangyayari yung computing and hindi sa processor. Your GPU will handle all the lifting much better if NVIDIA card gamit mo, the better the GPU the faster the response.
Understanding the AI Model and quantization.
Example :
FP16 (Original 16-bit): Crisp, pristine quality, but the file is massive (e.g., ~14 GB for a 7B model).
8-bit (Q8_0): Virtually identical quality to FP16, but cuts memory usage in half (~7.5 GB)
4-bit (Q4_K_M): 1080p equivalent—slight drop in fine detail, but uses 70% less VRAM (~4.5 GB).
2-bit (Q2_K): 480p equivalent—blurry, highly compressed, and prone to losing logic or factual details.
Lets say "Gemma 4 26B A4B QAT" yung napili mong model.
26B - Means it has 26 billion parameters. Parang ito yung overall brain capacity nung AI model.
A4B - Instead of puling 26 billion parameters (which is super bigat) the model smartly pulls 4 billion parameters only per token. Bali yung have the brain of 26 billion but with a speed of 4 billion model.
QAT - Medyo malalim na to eh, pero in short word eto yung training model na ginamit.