Project Overview
Built an encoder-decoder bilingual translation system for Jeju dialect and standard Korean using KoT5/KoBART with PEFT (LoRA, QLoRA) for low-resource training environments.
Problem Definition & Goals
- Problem: Translation quality for Jeju dialect is hard to sustain under limited GPU resources while preserving linguistic nuance.
- Goal 1: Build a single model that supports both dialect-to-standard and standard-to-dialect translation.
- Goal 2: Compare Full fine-tuning, LoRA, and QLoRA for quality-efficiency trade-offs.
- Goal 3: Achieve practical quantitative performance under constrained hardware.
Key Features & Contributions
- Dataset Engineering: Constructed 453k parallel pairs from AIHub Jeju dialect data and removed identical pairs to increase linguistic signal density.
- Tag-based Multi-task Learning: Introduced <dialect_to_standard> and <standard_to_dialect> special tokens for bidirectional translation in one model.
- Training Pipeline: Ran controlled Full/LoRA/QLoRA experiments with Seq2SeqTrainer.
- Quantitative Evaluation: Validated quality with BLEU and ROUGE-L metrics.
- Final Performance: Reached BLEU 0.56 and ROUGE-L 0.60 with stable translation quality.
Technical Challenges & Solutions
- OOM Constraint: Full fine-tuning repeatedly failed on Colab T4 (16GB) due to optimizer-state memory overhead.
- Memory Optimization: Adopted 4-bit QLoRA to drastically reduce memory usage while preserving quality.
- Ablation Validation: Compared Full and QLoRA with additional compute to verify quality parity.
- Outcome: Confirmed QLoRA delivers near-Full performance with much lower memory footprint.
Results & Learnings
- Result: Successfully deployed a single-model bidirectional translation workflow.
- Result: Demonstrated production-relevant translation quality in low-spec GPU environments.
- Insight: Reconfirmed structural strengths of encoder-decoder models for translation-heavy tasks.
- Next: Plan quantitative comparison against decoder-only models at similar parameter scales.