Inference

Inference is the production phase of AI — when trained models do actual work. Understanding inference helps explain why AI responses take time, cost money, and vary in speed.

Training vs Inference

Training:

• Happens once (or periodically)
• Requires massive compute
• Takes days to months
• Creates the model's capabilities

Inference:

• Happens every time you use AI
• Requires less compute (but still significant)
• Takes seconds
• Uses the trained model to generate output

Why Inference Matters for Vibe Coding

Speed:

• Larger models = slower inference
• More context = more to process
• Streaming shows results as they generate

Cost:

• You pay per token processed
• Input tokens + output tokens = total cost
• Complex prompts cost more

Quality:

• Better models often have slower inference
• Trade-off between speed and capability

Local vs Cloud Inference

Cloud inference (most common):

• Powerful hardware on provider's servers
• No local setup required
• Usage-based pricing

Local inference:

• Runs on your machine
• Privacy benefits
• Limited by your hardware
• Often smaller, less capable models

Most vibe coders use cloud inference through tools like Cursor, benefiting from powerful models without managing infrastructure.