Workshop 4: Training & Fine-Tuning Duration: 60 minutes | Level: Intermediate | Prerequisites: Workshops 1-3
What You’ll Master Transform your expert from generic to specialized by training it with your own data and knowledge.
Understanding Training
Learn the different types of training available
Document Upload
Add knowledge through documents and files
QA Pair Creation
Create question-answer pairs for precise training
Model Fine-Tuning
Fine-tune a base model with your data
Training Evaluation
Test and evaluate your training results
Types of Training B-Bot offers multiple training approaches:
┌────────────────────────────────────────────────────────────────┐ │ TRAINING METHODS │ ├────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Document │ │ QA │ │ Fine- │ │ │ │ Retrieval │ │ Pairs │ │ Tuning │ │ │ │ (RAG) │ │ │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ Add searchable Exact Q&A Train custom │ │ knowledge base matching model weights │ │ │ └────────────────────────────────────────────────────────────────┘
Document Retrieval (RAG) Best for: Large knowledge bases, manuals, documentationYour expert searches documents to find relevant information.
QA Pairs Best for: Specific questions, exact answers, brand voiceDirect question-answer mappings for precise responses.
Fine-Tuning Best for: Unique behavior, consistent style, specialized tasksTrain a custom model with your data.
Document Training (RAG) How RAG Works User Question: "What's the warranty on SmartHub?" │ ▼ ┌─────────────────┐ │ Vector Search │ ──► Finds relevant chunks │ in Documents │ from your uploaded docs └─────────────────┘ │ ▼ ┌─────────────────┐ │ LLM Generates │ ──► Creates answer using │ Response │ retrieved context └─────────────────┘ │ ▼ "The SmartHub has a 2-year warranty covering..."
Uploading Documents
Navigate to Training in the sidebar
Click Documents tab
Click Upload Documents
Format Best For Notes PDF Manuals, reports Extracts text and structure DOCX Word documents Preserves formatting TXT Plain text Simplest format MD Markdown docs Great for technical docs CSV Structured data Creates searchable rows JSON API docs, structured Maintains hierarchy
🎯 Exercise: Document Upload Create a simple product manual for training:
Create Document
Create a file called product_manual.md: # SmartHub Pro User Manual ## Quick Start Guide 1. Plug in your SmartHub Pro 2. Wait for the blue LED to blink 3. Open the TechGadgets app 4. Follow the on-screen setup ## Troubleshooting ### Device won't connect - Ensure WiFi is 2.4GHz - Move closer to router - Restart the device ### LED is red - Check power connection - Try different outlet - Contact support if persists ## Warranty Your SmartHub Pro includes a 2-year warranty covering manufacturing defects. Does not cover physical damage or water exposure.
Upload to B-Bot
Upload this document to your TechSupport AI expert
Test Retrieval
Ask: “What should I do if the LED is red?”
Document Processing Options
How documents are split for search:
Paragraph : Best for structured documentsSentence : Best for FAQsToken-based : Best for long documents
How text is converted to vectors:
OpenAI ada-002 : High quality, standardCohere : Good for multilingualLocal : Privacy-focused
How much context is shared between chunks:
Higher overlap = better context preservation
Lower overlap = faster search
QA Pair Training When to Use QA Pairs
✅ Great For
Brand-specific terminology
Exact pricing/policies
Consistent answers to common questions
Company voice and tone
⚠️ Less Effective For
Open-ended questions
Complex reasoning
Large knowledge bases
Frequently changing info
Creating QA Pairs Navigate to Training → QA Pairs :
🎯 Exercise: Create QA Pairs Create these QA pairs for your TechSupport AI:
QA Pair 1
QA Pair 2
QA Pair 3
Question: What are your support hours?
Answer: Our support team is available Monday through Friday, 9 AM to 6 PM EST. For urgent issues outside these hours, please email [email protected] and we'll respond within 4 hours.
Question: How do I return a product?
Answer: To initiate a return: 1. Log into your account at techgadgets.com/returns 2. Select the order containing the item 3. Click "Start Return" and follow the instructions 4. Print the prepaid shipping label 5. Ship within 30 days of purchase Refunds are processed within 5-7 business days.
Question: Do you offer international shipping?
Answer: Yes! We ship to over 40 countries. International shipping rates and delivery times vary by location. You can see exact costs at checkout before completing your order. Note: Some products may have shipping restrictions due to local regulations.
QA Pair Best Practices
Use questions as real users would ask them:
❌ “Provide information about shipping policies”
✅ “How long does shipping take?”
Add multiple phrasings for the same question:
“What’s the warranty?”
“How long is my product covered?”
“Is this under warranty?”
Provide full, helpful answers:
Include all relevant details
Add next steps or links
Use your brand voice
Fine-Tuning What is Fine-Tuning? Fine-tuning trains the model’s neural network weights with your data, creating a specialized version of the base model.
┌─────────────────────────────────────────────────────────┐ │ FINE-TUNING PROCESS │ ├─────────────────────────────────────────────────────────┤ │ │ │ Base Model (GPT-4o-mini) │ │ │ │ │ ▼ │ │ ┌──────────────────────┐ │ │ │ Your Training Data │ │ │ │ - QA pairs │ │ │ │ - Conversations │ │ │ │ - Examples │ │ │ └──────────────────────┘ │ │ │ │ │ ▼ │ │ Your Custom Model │ │ (Specialized behavior, your brand voice) │ │ │ └─────────────────────────────────────────────────────────┘
Starting a Fine-Tune Job
Prepare Data
Collect at least 50-100 high-quality training examples
Navigate to Fine-Tuning
Go to Training → Fine-Tuning
Select Base Model
Choose the model to fine-tune (e.g., GPT-4o-mini)
Upload Training Data
Upload your prepared dataset
Start Training
Begin the fine-tuning job and monitor progress
Fine-tuning uses JSONL format:
{ "messages" : [{ "role" : "system" , "content" : "You are TechSupport AI..." }, { "role" : "user" , "content" : "My device won't turn on" }, { "role" : "assistant" , "content" : "I'm sorry to hear that! Let's troubleshoot together..." }]} { "messages" : [{ "role" : "system" , "content" : "You are TechSupport AI..." }, { "role" : "user" , "content" : "How do I update firmware?" }, { "role" : "assistant" , "content" : "Great question! Here's how to update..." }]}
Fine-Tuning Tips
Data Quality Quality over quantity. 50 excellent examples beat 500 mediocre ones.
Diverse Examples Include various topics, question types, and edge cases.
Consistent Format Maintain consistent response style across all examples.
Iterate Fine-tune in rounds, testing and improving each time.
Model Distillation What is Distillation? Distillation transfers knowledge from a powerful model (teacher) to a smaller, faster model (student).
┌─────────────────────────────────────────────────────────┐ │ MODEL DISTILLATION │ ├─────────────────────────────────────────────────────────┤ │ │ │ Teacher Model (GPT-4o) │ │ - Powerful but expensive │ │ - Generates training examples │ │ │ │ │ ▼ │ │ Student Model (GPT-4o-mini) │ │ - Learns from teacher's outputs │ │ - Faster and cheaper │ │ - Similar quality for your use case │ │ │ └─────────────────────────────────────────────────────────┘
Benefits Aspect Before Distillation After Distillation Cost $$$$ (GPT-4o) $$ (fine-tuned mini) Speed ~3s per response ~1s per response Quality Excellent Very Good (for your domain)
Evaluating Training Results Testing Your Trained Expert
Create Test Set
Prepare 10-20 questions your expert should answer well
Run Tests
Ask each question and record the response
Evaluate
Score responses for accuracy, tone, and completeness
Iterate
Add more training data where gaps exist
Evaluation Criteria Criterion What to Check Accuracy Is the information correct? Completeness Is the answer thorough? Tone Does it match your brand voice? Relevance Does it answer the actual question? Helpfulness Would a real user be satisfied?
🎯 Challenge: Complete Training Pipeline
Upload 3 Documents
Add product manuals, FAQs, and policy documents
Create 10 QA Pairs
Cover common questions with perfect answers
Test with 5 Questions
Verify the expert uses the training data
Refine
Improve based on test results
Best Practices Summary
Start with RAG Document retrieval is fastest to implement and easiest to update
Add QA for Precision Use QA pairs for questions that must have exact answers
Fine-Tune Last Only fine-tune when you have enough data and clear improvement goals
Test Continuously Regular testing catches regressions early
Next Steps 
