Workshop 4: Training & Fine-Tuning

Duration: 60 minutes | Level: Intermediate | Prerequisites: Workshops 1-3

What You’ll Master

Transform your expert from generic to specialized by training it with your own data and knowledge.

Understanding Training

Learn the different types of training available

Document Upload

Add knowledge through documents and files

QA Pair Creation

Create question-answer pairs for precise training

Model Fine-Tuning

Fine-tune a base model with your data

Training Evaluation

Test and evaluate your training results

Types of Training

B-Bot offers multiple training approaches:

┌────────────────────────────────────────────────────────────────┐
│                    TRAINING METHODS                            │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐      │
│   │   Document   │   │     QA       │   │    Fine-     │      │
│   │   Retrieval  │   │    Pairs     │   │   Tuning     │      │
│   │   (RAG)      │   │              │   │              │      │
│   └──────────────┘   └──────────────┘   └──────────────┘      │
│         │                  │                  │               │
│         ▼                  ▼                  ▼               │
│   Add searchable     Exact Q&A          Train custom         │
│   knowledge base     matching           model weights        │
│                                                                │
└────────────────────────────────────────────────────────────────┘

Document Retrieval (RAG)

Best for: Large knowledge bases, manuals, documentationYour expert searches documents to find relevant information.

QA Pairs

Best for: Specific questions, exact answers, brand voiceDirect question-answer mappings for precise responses.

Fine-Tuning

Best for: Unique behavior, consistent style, specialized tasksTrain a custom model with your data.

Document Training (RAG)

How RAG Works

User Question: "What's the warranty on SmartHub?"
           │
           ▼
   ┌─────────────────┐
   │ Vector Search   │ ──► Finds relevant chunks
   │ in Documents    │     from your uploaded docs
   └─────────────────┘
           │
           ▼
   ┌─────────────────┐
   │  LLM Generates  │ ──► Creates answer using
   │  Response       │     retrieved context
   └─────────────────┘
           │
           ▼
"The SmartHub has a 2-year warranty covering..."

Uploading Documents

Navigate to Training in the sidebar
Click Documents tab
Click Upload Documents

Supported Formats

Format	Best For	Notes
PDF	Manuals, reports	Extracts text and structure
DOCX	Word documents	Preserves formatting
TXT	Plain text	Simplest format
MD	Markdown docs	Great for technical docs
CSV	Structured data	Creates searchable rows
JSON	API docs, structured	Maintains hierarchy

🎯 Exercise: Document Upload

Create a simple product manual for training:

Create Document

Create a file called product_manual.md:

# SmartHub Pro User Manual

## Quick Start Guide

1. Plug in your SmartHub Pro
2. Wait for the blue LED to blink
3. Open the TechGadgets app
4. Follow the on-screen setup

## Troubleshooting

### Device won't connect
- Ensure WiFi is 2.4GHz
- Move closer to router
- Restart the device

### LED is red
- Check power connection
- Try different outlet
- Contact support if persists

## Warranty

Your SmartHub Pro includes a 2-year warranty 
covering manufacturing defects. Does not cover
physical damage or water exposure.

Upload to B-Bot

Upload this document to your TechSupport AI expert

Test Retrieval

Ask: “What should I do if the LED is red?”

Document Processing Options

Chunking Strategy

How documents are split for search:

Paragraph: Best for structured documents
Sentence: Best for FAQs
Token-based: Best for long documents

Embedding Model

How text is converted to vectors:

OpenAI ada-002: High quality, standard
Cohere: Good for multilingual
Local: Privacy-focused

Overlap

How much context is shared between chunks:

Higher overlap = better context preservation
Lower overlap = faster search

QA Pair Training

When to Use QA Pairs

✅ Great For

Brand-specific terminology
Exact pricing/policies
Consistent answers to common questions
Company voice and tone

⚠️ Less Effective For

Open-ended questions
Complex reasoning
Large knowledge bases
Frequently changing info

Creating QA Pairs

Navigate to Training → QA Pairs:

🎯 Exercise: Create QA Pairs

Create these QA pairs for your TechSupport AI:

QA Pair 1
QA Pair 2
QA Pair 3

Question:

What are your support hours?

Answer:

Our support team is available Monday through Friday, 
9 AM to 6 PM EST. For urgent issues outside these hours,
please email [email protected] and we'll respond 
within 4 hours.

Question:

How do I return a product?

Answer:

To initiate a return:
Log into your account at techgadgets.com/returns
Select the order containing the item
Click "Start Return" and follow the instructions
Print the prepaid shipping label
Ship within 30 days of purchase

Refunds are processed within 5-7 business days.

Question:

Do you offer international shipping?

Answer:

Yes! We ship to over 40 countries. International shipping 
rates and delivery times vary by location. You can see 
exact costs at checkout before completing your order.

Note: Some products may have shipping restrictions 
due to local regulations.

QA Pair Best Practices

Write Natural Questions

Use questions as real users would ask them:

❌ “Provide information about shipping policies”
✅ “How long does shipping take?”

Include Variations

Add multiple phrasings for the same question:

“What’s the warranty?”
“How long is my product covered?”
“Is this under warranty?”

Complete Answers

Provide full, helpful answers:

Include all relevant details
Add next steps or links
Use your brand voice

Fine-Tuning

What is Fine-Tuning?

Fine-tuning trains the model’s neural network weights with your data, creating a specialized version of the base model.

┌─────────────────────────────────────────────────────────┐
│                    FINE-TUNING PROCESS                   │
├─────────────────────────────────────────────────────────┤
│                                                         │
│   Base Model (GPT-4o-mini)                             │
│        │                                                │
│        ▼                                                │
│   ┌──────────────────────┐                             │
│   │  Your Training Data  │                             │
│   │  - QA pairs          │                             │
│   │  - Conversations     │                             │
│   │  - Examples          │                             │
│   └──────────────────────┘                             │
│        │                                                │
│        ▼                                                │
│   Your Custom Model                                     │
│   (Specialized behavior, your brand voice)              │
│                                                         │
└─────────────────────────────────────────────────────────┘

Starting a Fine-Tune Job

Prepare Data

Collect at least 50-100 high-quality training examples

Navigate to Fine-Tuning

Go to Training → Fine-Tuning

Select Base Model

Choose the model to fine-tune (e.g., GPT-4o-mini)

Upload Training Data

Upload your prepared dataset

Start Training

Begin the fine-tuning job and monitor progress

Training Data Format

Fine-tuning uses JSONL format:

{"messages": [{"role": "system", "content": "You are TechSupport AI..."}, {"role": "user", "content": "My device won't turn on"}, {"role": "assistant", "content": "I'm sorry to hear that! Let's troubleshoot together..."}]}
{"messages": [{"role": "system", "content": "You are TechSupport AI..."}, {"role": "user", "content": "How do I update firmware?"}, {"role": "assistant", "content": "Great question! Here's how to update..."}]}

Fine-Tuning Tips

Data Quality

Quality over quantity. 50 excellent examples beat 500 mediocre ones.

Diverse Examples

Include various topics, question types, and edge cases.

Consistent Format

Maintain consistent response style across all examples.

Iterate

Fine-tune in rounds, testing and improving each time.

Model Distillation

What is Distillation?

Distillation transfers knowledge from a powerful model (teacher) to a smaller, faster model (student).

┌─────────────────────────────────────────────────────────┐
│                   MODEL DISTILLATION                     │
├─────────────────────────────────────────────────────────┤
│                                                         │
│   Teacher Model (GPT-4o)                               │
│   - Powerful but expensive                              │
│   - Generates training examples                         │
│        │                                                │
│        ▼                                                │
│   Student Model (GPT-4o-mini)                          │
│   - Learns from teacher's outputs                       │
│   - Faster and cheaper                                  │
│   - Similar quality for your use case                   │
│                                                         │
└─────────────────────────────────────────────────────────┘

Benefits

Aspect	Before Distillation	After Distillation
Cost	$$$$ (GPT-4o)	$$ (fine-tuned mini)
Speed	~3s per response	~1s per response
Quality	Excellent	Very Good (for your domain)

Evaluating Training Results

Testing Your Trained Expert

Create Test Set

Prepare 10-20 questions your expert should answer well

Run Tests

Ask each question and record the response

Evaluate

Score responses for accuracy, tone, and completeness

Iterate

Add more training data where gaps exist

Evaluation Criteria

Criterion	What to Check
Accuracy	Is the information correct?
Completeness	Is the answer thorough?
Tone	Does it match your brand voice?
Relevance	Does it answer the actual question?
Helpfulness	Would a real user be satisfied?

🎯 Challenge: Complete Training Pipeline

Upload 3 Documents

Add product manuals, FAQs, and policy documents

Create 10 QA Pairs

Cover common questions with perfect answers

Test with 5 Questions

Verify the expert uses the training data

Refine

Improve based on test results

Best Practices Summary

Start with RAG

Document retrieval is fastest to implement and easiest to update

Add QA for Precision

Use QA pairs for questions that must have exact answers

Fine-Tune Last

Only fine-tune when you have enough data and clear improvement goals

Test Continuously

Regular testing catches regressions early

Next Steps

Workshop 5: DeepAgents

Unlock autonomous AI capabilities

Training Documentation

Detailed training reference

Getting Started

Experts

Chat

Training & Knowledge

Apps & Integrations

DeepAgents

Distribution

Account & Settings

Advanced Features

Declarations

​Workshop 4: Training & Fine-Tuning

​What You’ll Master

​Types of Training

Document Retrieval (RAG)

QA Pairs

Fine-Tuning

​Document Training (RAG)

​How RAG Works

​Uploading Documents

​Supported Formats

​🎯 Exercise: Document Upload

​Document Processing Options

​QA Pair Training

​When to Use QA Pairs

✅ Great For

⚠️ Less Effective For

​Creating QA Pairs

​🎯 Exercise: Create QA Pairs

​QA Pair Best Practices

​Fine-Tuning

​What is Fine-Tuning?

​Starting a Fine-Tune Job

​Training Data Format

​Fine-Tuning Tips

Data Quality

Diverse Examples

Consistent Format

Iterate

​Model Distillation

​What is Distillation?

​Benefits

​Evaluating Training Results

​Testing Your Trained Expert

​Evaluation Criteria

​🎯 Challenge: Complete Training Pipeline

​Best Practices Summary

Start with RAG

Add QA for Precision

Fine-Tune Last

Test Continuously

​Next Steps

Workshop 5: DeepAgents

Training Documentation

Workshop 4: Training & Fine-Tuning

What You’ll Master

Types of Training

Document Training (RAG)

How RAG Works

Uploading Documents

Supported Formats

🎯 Exercise: Document Upload

Document Processing Options

QA Pair Training

When to Use QA Pairs

Creating QA Pairs

🎯 Exercise: Create QA Pairs

QA Pair Best Practices

Fine-Tuning

What is Fine-Tuning?

Starting a Fine-Tune Job

Training Data Format

Fine-Tuning Tips

Model Distillation

What is Distillation?

Benefits

Evaluating Training Results

Testing Your Trained Expert

Evaluation Criteria

🎯 Challenge: Complete Training Pipeline

Best Practices Summary

Next Steps