KARL: Knowledge-Augmented Reinforcement Learning for LLM Agents

Getting Started • Training • Evaluation • Citation

---

This repo is the code repository for KARL. KARL is a framework that enables LLM agents to dynamically explore structured knowledge sources through multi-turn interactions.

---

Our KARL framework supports stable and scalable multi-turn reinforcement learning, facilitating real-time knowledge access, long-horizon decision-making, and instant curiosity-based feedback.

Scores across 6 knowledge-intensive agentic datasets based on Knowledge Graph and DataBase.

---

Getting Started

Installation

git clone <this-repo>
cd KARL

# Set up the training environment
cd train
pip install -e .
pip install -r requirements.txt

# Set up the task environment
cd ../env
conda create -n karl python=3.9
conda activate karl
pip install -r requirements.txt

---

Repository Structure

KARL/
├── train/                # RL training framework (based on verl)
│   ├── configs/          #   Training configurations
│   ├── scripts/          #   Launch scripts
│   ├── eval/             #   Evaluation & model conversion tools
│   ├── verl/             #   Core training codebase
│   └── README.md         #   Training documentation
├── env/                  # Task environment server
│   ├── configs/          #   Task & service configurations
│   ├── src/              #   Environment source code
│   ├── data/             #   Task datasets
│   └── README.md         #   Environment documentation
└── README.md             # This file

Directory	Description
train/	Multi-turn RL training framework built on verl. Uses GRPO with asynchronous rollout and FSDP2 distributed training.
env/	Task environment server hosting interactive task workers (KG, DBBench, etc.) and a controller for agent interaction during training and evaluation.

---

Training

1. Start the task environment — see env/README.md for detailed setup (KG service, Docker for DBBench, etc.).

2. Launch RL training — see train/README.md for configuration and launch instructions.

cd train
export MLP_WORKER_NUM=<NUM_NODES>
export MLP_GPU=8
bash scripts/start_agentbench.sh example

---

Evaluation

Post-training evaluation tools are located in train/eval/. See train/README.md for details.

cd train

# API-based evaluation
python eval/server_agent_multitask.py   -m <model_name>   -u <api_url>   -c <controller_url>   <task_name>

# Compute metrics (avg score, Pass@K, etc.)
python eval/check.py <output.jsonl>

# Convert FSDP checkpoint to HuggingFace format
python eval/verl2hf.py --local_dir <ckpt_dir> --output_dir <hf_dir>

---

Citation

@inproceedings{sun-etal-2026-karl,
    title = "{KARL}: Reinforcement Learning for {LLM} Agents on Multi-Turn Knowledge-Intensive Agentic Tasks",
    author = "Sun, Xueqiao  and
      Liu, Xiao  and
      Lv, Bowen  and
      Zhang, Hanchen  and
      Jing, Bohao  and
      Qi, Zehan  and
      Xu, Yifan  and
      Dong, Yuxiao  and
      Tang, Jie",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.2196/",
    doi = "10.18653/v1/2026.acl-long.2196",
    pages = "47539--47558",
    ISBN = "979-8-89176-390-6",
    abstract = "Large Language Models have shown remarkable potential as autonomous agents, but their effectiveness in knowledge-intensive tasks remains limited by passive knowledge utilization. We introduce KARL (Knowledge-Augmented Reinforcement Learning), a framework that enables LLM agents to dynamically explore structured knowledge sources through multi-turn interactions. Unlike existing retrieval-augmented approaches, KARL empowers agents to proactively decide when and what knowledge to acquire during task execution. Our framework incorporates online reinforcement learning with curiosity-driven reward shaping, explicitly incentivizing knowledge exploration while optimizing tool-use behaviors end-to-end. Extensive evaluation across six structured knowledge benchmarks demonstrates that KARL achieves state-of-the-art performance, with our Qwen2.5-14B-based agent significantly outperforming GPT-4o, Claude-4, and o4-mini on both knowledge graph and database tasks.Source code is available at https://github.com/THUDM/KARL."
}

---

License

This project is licensed under the Apache License 2.0.

Acknowledgements

• Training framework built on verl