Evaluating and Debugging Generative Al
Disclaimer
This note contains only the slides, no personal notes.
Outline
- Instrument W&B (Weights and Biases) in an ML training pipeline
- Training diffusion models
- Evaluating diffusion models
- Evaluating LLMs
- Fine-tuning LLMs
Comparing model outputs
Model Registry
It is a central system of records for the models.
- Publish production-ready models
- Move model versions through the lifecycle from staging to production.
- Collaborate on models across teams.
- Audit model lineage across training, evaluation, and production
- Automate downstream actions
Training & Finetuning LLMs
Training from scratch
- Long & expensive training runs
- Expensive & difficult evaluations
- Monitoring is critical
- Ability to restore training from a checkpoint
Fine-tuning
- Efficient methods being developed
- Expensive & difficult evaluations
