Nemotron-Research-Tool-N1: Tool-Using Language Models with Reinforced Reasoning

1. Introduction

In recent years, equipping large language models (LLMs) with external tools or functions has attracted significant attention, demonstrating impressive performance across a wide range of domains  [1,29]. For instance, LLMs augmented with search engines can answer questions that go beyond their training data [10], while those equipped with Python interpreters are capable of solving complex mathematical problems by leveraging external libraries [38]. These tools effectively enable LLMs to operate beyond purely textual tasks, substantially extending their functional capabilities.

To enhance LLMs tool-calling capability, existing research has primarily focused on synthesizing large volumes of tool-use trajectories using advanced language models [16,43,47], followed by supervised fine-tuning (SFT) on the generated data. Although step-by-step reasoning has been shown to play a critical role in enabling LLMs to solve complex tasks [37], these synthetic datasets often lack explicit reasoning steps. Consequently, current pipelines typically supervise only the tool calls step, without providing guidance on the underlying reasoning process during training. In many cases, reasoning is either omitted entirely in training stage or deferred to the inference stage via prompting techniques [41,42]. While one workaround is to distill reasoning trajectories from advanced models and then train student models via SFT [2], this approach often yields pseudo reasoning: models merely learn to mimic surface-level patterns from expert demonstrations without truly internalizing the decision-making process [2].

Most recently, DeepSeek R1 [6] has demonstrated that simple rule-based reinforcement learning can substantially improve the complex reasoning capabilities of LLMs [18,19,31]. In the R1-style RL training, rewards are assigned based on the correctness of the final answer and reasoning format, allowing the model to learn the intermediate reasoning steps without explicit reasoning supervision. This paradigm naturally inspired us to explore the utilization of R1-style RL in training tool call models, for the following reasons: (1) Rule-based reward design provides interpretable supervision signals by verifying the correctness of tool calls. In contrast to SFT, which relies on exact next-token prediction and enforces strict output matching, RL allows for more flexibility. Semantically equivalent tool calls, such as those with reordered arguments, can still be rewarded correctly. This enables models to generalize beyond rigid string-level imitation. (2) R1-style reinforcement learning does not require tool-call data with explicit reasoning annotations. Instead, it imposes minimal constraints, requiring only that the model follow a structured output format. This makes it possible to train reasoning capabilities directly from available SFT data. As a result, the model can acquire reasoning skills without curating reasoning traces.

Figure 1. Overview of the training pipeline for Nemotron-Research-Tool-N1 (Tool-N1). Starting from standard SFT tool-calling data comprising user queries and candidate tools, we train LLMs to produce structured reasoning and tool calls using a binary reward function within the GRPO algorithm. As supervision is only applied to the format and tool-call correctness, the training process does not require curated reasoning trajectories.

Figure 1. Overview of the training pipeline for Nemotron-Research-Tool-N1 (Tool-N1). Starting from standard SFT tool-calling data comprising user queries and candidate tools, we train LLMs to produce structured reasoning and tool calls using a binary reward function within the GRPO algorithm. As supervision is only applied to the format and tool-call correctness, the training process does not require curated reasoning trajectories.

Building on these insights, we introduce Nemotron-Research-Tool-N1 (Tool-N1)1, a series of tool-using language models trained with a rule-based reinforcement learning. The training procedure enforces a simple-tructured reasoning-action format, guiding the model to produce explicit reasoning before invoking tools. We employ a binary reward function that evaluates both the structural correctness of the reasoning format and the accuracy of tool invocations. This reward design provides precise yet flexible supervision, allowing variation in argument ordering while ensuring functional correctness. We conduct extensive experiments on BFCL [39] and API-Bank [12]. Empirical results show that our models, Tool-N1-7B and Tool-N1-14B-built upon Qwen2.5-7B/14B-Instruct consistently outperform GPT-4o and other strong baselines across both benchmarks.

2. Related Work

Tool Learning. Enhancing large language models (LLMs) with external tools has demonstrated significant potential in addressing complex tasks [1,29,36]. Typical applications include integrating LLMs with search engines [10,11,32,49], calculator [24], vision tools [7,20], and Python interpreters [33,35,38]. Research efforts aimed at improving LLMs’ tool-use capabilities can be broadly divided into two categories. The first one focuses on improving the LLM itself. This typically involves curating large-scale supervised datasets [16,21,28,43,47] and applying either SFT [16,28,47] or DPO reinforcement learning [44,45]. These approaches are primarily data-centric [46] and rely heavily on the availability of high-quality, large-scale training data. The second category explores prompting-based techniques that do not alter the underlying model. These methods aim to elicit tool-use behavior through strategies such as in-context learning with demonstrations [9,26] or the design of advanced prompting schemes [41,42,48]. However, due to the inherent limitations of the base models, such prompting approaches often struggle with tasks that require intricate reasoning or complex multi-step actions.

LLM Reasoning and Reinforcement Learning. Recent efforts in the LLM research community have increasingly focused on improving reasoning capabilities, marking a shift from train-time scaling to test-time scaling [23], with the goal of enabling models to handle complex, multi-step problem-solving tasks [37]. Earlier approaches often relied on step-level supervision or learned reward models to guide the model’s reasoning trajectory [5,13]. More recently, DeepSeek-R1 [6] has demonstrated that simple rule-based reinforcement learning can effectively induce strong reasoning behaviors. In this framework, reward functions are defined based on predefined criteria, i.e., checking whether the model’s final answer matches the ground truth in math problems. These rewards target only the correctness of the final answer, allowing the model to learn the intermediate reasoning steps without explicit reasoning supervision. This R1-style reasoning paradigm has shown success across various domains, including mathematics [30], coding [15], and vision tasks [31]. In parallel, several recent studies have integrated external tools such as search engines [8] and code interpreters [3] into reinforcement learning frameworks for LLM reasoning. These approaches primarily focus on training LLMs to effectively utilize a single, general-purpose tool to support one single reasoning chain. In contrast, our work aims to enhance LLMs’ tool-calling ability in more general tool-use settings, where multiple tools with complex functionalities and argument structures are available.

3. Problem Formulation

We first provide the problem formulation. Consider a large language model (LLM) and a set of external tools $\mathcal{Z}={\left\{z_i\right\}}_{i=i}^I$ that the LLM can access. Each tool $z_i$ could be represented as a 3-tuple $(n_i,d_i,k_i)$ which includes essential information for tool usage: $n_i$ denotes the tool’s name, $d_i$ provides a natural language descriptions of the tool, and $k_i$ specifies the tool’s input parameters instructions. The model’s objective is to respond to user queries in accordance with a policy $\pi$. To achieve this, the LLM may issue multiple tool calls throughout the interaction, each with appropriate parameters.

At any decision step t, the LLM receives two types of input: (1) the historical context $c_t$, which consists of all preceding tool-call and observation pairs, and (2) the set of currently available tools $\mathcal{Z}$ that can be utilized at this step. The LLM must then decide on the next action. Formally, the decision-making process is defined as:

$$\pi (c_t,\tilde{\mathcal{Z}})\to a_t,\mathrm{s}.\mathrm{t}.a_t\subseteq \mathcal{Z},$$

(1)

where $a_t$ denotes the action selected at step t. This action corresponds to one or more tool calls drawn from the accessible tool subset $\tilde{\mathcal{Z}}$. $c_t$ represents the historical context. Specifically,

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill a_t& =\{z_0\left(p_0\right),\dots ,z_m\left(p_m\right)\},\hfill \\ \hfill c_t& =(a_0,o_0,\dots ,a_t,o_t)\hfill \end{array}\right.\end{array}$$

(2)

where each $z_m$ denotes the m-th tool invoked and $p_m$ its corresponding parameters. The value of m indicates the number of tool calls made at time t. We employ $o_t$ denotes the observation after action $a_t$ is taken. The ultimate goal of tool learning is to enable LLMs with a generalized policy $\pi$ that effectively addresses user queries by producing a coherent sequence of action-observation pairs $(a_t,o_t)$.

4. Nemotron-Research-Tool-N1

In this section, we introduce Nemotron-Research-Tool-N1, an R1-style reinforcement learning framework designed to train general-purpose tool-using language models. Tool-N1 is built upon the GRPO reinforcement learning algorithm [6,30], aiming to improve the model’s tool-calling capabilities in complex scenarios where the LLM is tasked with solving queries using an accessible set of tools.

Formally, given the historical context $c_t$ and the set of currently available tools $\mathcal{Z}$, the model generates a set of candidate responses $[O^1,O^2,...,O^N]$. Each response consists of (1) textual reasoning and (2) an associated action $a_n$. These responses are evaluated using a reward function, yielding a reward set $\{r_1,r_2,...,r_N\}$. The GRPO algorithm is then employed to estimate advantages and update the policy model, subject to KL divergence constraints. The relative advantage $A_i$ of the i-th response is computed as follows:

$$A_i={\displaystyle \frac{r_i-\mathrm{mean}\left(\{r_1,r_2,\dots ,r_N\}\right)}{\mathrm{std}\left(\{r_1,r_2,\dots ,r_N\}\right)}},$$

(3)

where mean and std represent the mean and standard deviation of the rewards, respectively. In the subsequent sections, we detail the following components of our approach: (1) the preparation of supervised fine-tuning data and its integration into the R1-style RL training pipeline, (2) the structured reasoning template used during RL training, and (3) the reward modeling strategy employed to guide the learning process.

4.1. Data Preparation

Numerous prior works have focused on collecting large-scale tool-calling trajectories [16,28,43,47], followed by supervised fine-tuning (SFT) to improve the tool-use capabilities of LLMs. These datasets typically consist of a natural language user query Q, paired with a sequence of ground-truth tool invocation steps in the form $(a_0,o_o,\dots ,a_t,o_t)$. The model is then trained to predict each subsequent action $a_t$, conditioned on the observed trajectory up to that point. However, SFT often exhibits limited generalization ability, as the model tends to memorize the training trajectories rather than developing robust, intrinsic reasoning capabilities for tool usage.

To fully leverage the available SFT data in the community, we unify and preprocess data from xLAM [47] and a subset of ToolACE [16], which provide single-turn and multi-turn synthetic tool-calling trajectories. As these datasets are generated by potentially unstable LLMs, they often contain inconsistencies and unstructured formats unsuitable for GRPO training. We standardize the dataset by filtering out samples with invalid tool calls, specifically, those involving tools absent from the candidate tool list. Available tools are extracted from the system prompt, and both candidate tools and ground-truth tool calls are parsed into structured dictionary formats. Instances that fail JSON parsing or contain formatting inconsistencies are discarded. This preprocessing yields a clean and consistent dataset suitable for reinforcement learning. For multi-turn data from the ToolACE subset, we further segment each trajectory into multiple single-step prediction instances, where each instance contains one target tool call and the preceding steps are treated as context. We train LLMs using R1-style GRPO to predict each tool invocation step based on this contextual information and provided tools. We present the statistic information of the proceed data in Table 1.

4.2. Thinking Template

Following Guo et al. [6], we adopt a lightweight prompting template to elicit tool calls from the LLM which is shown in Figure 2. The prompt explicitly instructs the model to generate intermediate reasoning within <think>...</think> tags, followed by the tool invocation enclosed in <tool_call>...</tool_call> tags. The design philosophy behind this template is to minimize reliance on overly rigid formatting rules, which could reducing the risk of overfitting to specific prompt patterns. By allowing greater flexibility in how the model expresses its reasoning, we aim to promote more robust generalization across diverse tool-use scenarios. Additionally, the use of this lightweight prompting design during training enables the resulting model to be more easily integrated with more sophisticated prompting strategies [41,42,48].

4.3. Reward Modeling

Following the data preparation described in Section 4.1, we construct a training dataset in which each ground-truth tool call is represented as a structured dictionary. This format enables reliable verification of tool names and argument-value pairs during reinforcement learning rather than simply string match. Leveraging this structure, we define a binary reward function in the R1-style, which jointly evaluates the correctness of the reasoning format and the accuracy of the tool call, including its name and arguments.

Formate Checking. Following prior work [6,22,30], we incorporate format checking during training to verify whether the model’s output adheres to the expected structural conventions—specifically, whether the reasoning is enclosed within <think>...</think> tags and the tool call is properly placed within <tool_call>...</tool_call> tags. This structural constraint encourages the model to engage in explicit reasoning before tool invocation, rather than shortcutting to the final answer. By enforcing format adherence, we aim to cultivate the model’s intrinsic reasoning ability, which may potentially contributes to improved generalization—particularly on out-of-distribution inputs.

Tool-Calling Checking. We also check the correctness of the tool call itself. The tool-call output is parsed as a dictionary, enabling exact matching against the ground-truth call. This involves checking whether the predicted tool name matches the ground-truth and whether all required arguments are present with correct values. This strict matching criterion ensures that the model learns to generate functionally precise and executable tool calls. Compared to the next-token prediction logic in SFT, this dictionary-based matching introduces greater flexibility. It allows argument order to vary without penalization, encouraging the model to focus on the underlying semantics of tool invocation rather than surface-level memorization. This design promotes a deeper understanding of tool usage and supports better generalization.

Binary Reward Definition. Given a context $c_t$ and the predicted action $a_t$, we define a binary reward function $r(c_t,a_t)\in \{0,1\}$ that assigns a reward of 1 if both of the following conditions are met: (1) Format Correctness: The model output adheres to the structural format, i.e., contains both <think>...</think> and <tool_call>...</tool_call> tags. (2) Tool Call Correctness: The predicted tool call $a_t$ exactly matches the ground-truth call $a_t^{*}$ in both tool name and all argument key-value pairs.

$$\begin{array}{c}\hfill r(c_t,a_t)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathtt{FormatCorrect}\left(a_t\right)\wedge \mathtt{ToolCallMatch}(a_t,a_t^{*})\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(4)

where $\mathtt{FormatCorrect}\left(a_t\right)$ returns true if the output is correctly wrapped in both required tags, and the $\mathtt{ToolCallMatch}(a_t,a_t^{*})$ returns true if $a_t$ matches the ground-truth tool call $a_t^{*}$ exactly in structure and content.

5. Experiments

We conduct experiments to prove the superiority of the proposed method. We begin by providing the experimental settings in Section 5.1. We then evaluate the training method on two typical benchmarks to verify its effectiveness in Section 5.2. Finally, we perform in-depth investigations of the proposed method.

5.1. Settings

Datasets. We primarily utilize subsets of ToolACE [16] and xLAM [47] as our training dataset. ToolACE [16] encompasses a wide range of tool-calling scenarios, including examples with multiple candidate tools and parallel function calls, and covers a pool of 26,507 diverse tools. In contrast, xLAM focuses on single-turn function calling, comprising 60,000 instances collected through APIGen [17].

Models. Unless otherwise noted, we use Qwen2.5-7B/14B-Instruct [40] as the primary backbone model throughout this work. To assess the generalization ability of our method, we also perform evaluations on alternative backbone models, including multiple variants from the LLaMA family across different scales. In our experiments, we compare against both general-purpose open-source models, such as the GPT series and Gemini-2.0, as well as specialized tool-calling models, including ToolACE-8B [16], xLAM-2 [27], and Hammer2.1 [14].

Benchmarks. We primarily evaluate performance on single-turn tool-calling queries. We evaluate our approach on several representative benchmarks, including the Berkeley Function Call Leaderboard (BFCL) [39] and API-Bank [12]. For BFCL, we conduct evaluations on both the Non-live and Live subsets, corresponding to synthetic and real-world data, respectively. Each subset includes four categories: Simple, Multiple, Parallel, and Parallel Multiple. Simple and Multiple scenarios both involve the invocation of a single tool, with the Multiple category featuring multiple candidate tools. In contrast, Parallel and Parallel Multiple scenarios require the simultaneous invocation of multiple tools. For API-Bank, we exclude multi-turn cases from our evaluation. Performance across all benchmarks is reported in terms of accuracy.

Other Implementation Details. We conduct all experiments using the open-source reinforcement learning library Verl. Training is performed with a batch size of 1024 and a learning rate of $1\times {10}^{-6}$. The temperature is fixed at 0.7. We set the entropy coefficient to 0, as we observe that introducing entropy negatively impacts exploration during training. The KL divergence loss coefficient is set to $1\times {10}^{-3}$ across all experiments. All training runs are executed on a cluster of 4 nodes, each equipped with 8 NVIDIA H100 80GB GPUs.

5.2. Main Results

Results on BFCL.Table 2 presents the evaluation results on the BFCL benchmark, including detailed accuracy scores for each subcategory. We use the official evaluation script from the BFCL leaderboard and report average accuracy across categories. The results clearly demonstrate that all the Tool-N1-7B/14B models achieving the best overall performance, outperforming both state-of-the-art closed-source models, such as GPT-4o, and specialized fine-tuned models, including xLAM-2-70B and ToolACE-8B. Notably, the trained tool-call reasoning model significantly outperforms supervised fine-tuning baselines trained on the same data sources (i.e., the ToolACE [16] and xLAM [47] series). The results prove that R1-style reinforcement learning offers a more effective paradigm for enhancing the tool-calling capabilities of LLMs compared to standard supervised fine-tuning.

Results on API-Bank. Additionally, to provide a more comprehensive evaluation, we conduct experiments on the API-Bank benchmark [12], with results shown in Table 3. The Tool-N1-7B and Tool-N1-14B models consistently outperform the baselines across most cases. Notably, Tool-N1-7B/14B achieve 4.12% and 5.03% higher accuracy than GPT-4o, respectively, clearly demonstrating the effectiveness of the method.

5.3. Deep Analysis

5.3.1. Scalability and Generalizability

Scalability. The scaling law, which characterizes the relationship between model size and performance, plays a critical role in understanding the effectiveness of the training methods. We assess the scaling behavior of the proposed training method by evaluating a range of model sizes, including 0.5B, 1.5B, 3B, 7B and 14B from the Qwen2.5-Instruct series. For comparison, we also report the performance of the original instruction-tuned models without any additional training. We report average performance on both the Live and Non-Live categories of the BFCL benchmark, with detailed results presented in Figure 4. As expected, larger models consistently outperform smaller ones in both evaluation settings. Notably, performance improvements from post-training are limited for smaller models (0.5B and 1.5B), whereas larger models exhibit substantial gains. These findings suggest that the R1-style training method scales more effectively with increasing model size.

Generalizability. We further evaluate the impact of different backbone LLMs [34] to investigate the generalization capability of the proposed training method. In addition to the Qwen series, we include experiments using LLaMA-based models: LLaMA3-8B-Instruct and LLaMA3.1-8B-Instruct. These evaluations are conducted on the BFCL benchmark, with results presented in Figure 5. Our findings show that Qwen2.5-Instruct significantly outperforms both LLaMA variants at the same model scale. This advantage is likely due to Qwen’s inherently stronger reasoning capabilities, as previously observed by Gandhi et al. [4]. As a result, the R1-style training paradigm is able to elicit better performance when applied to Qwen.

5.3.2. Ablations

Ablations on Reward Design. To assess how reward granularity affects model behavior, we evaluate Tool-N1-7B under two reward schemes: fine-grained and binary (Table 3). The fine-grained setting provides partial reward, 0.2 for correct reasoning format and an additional 0.2 for matching function names, even if the final function call is incorrect. In contrast, the binary setting only gives a reward of 1.0 when all components are correct, including reasoning, function name, and arguments. Tool-N1 achieves consistently better performance with binary rewards, particularly on the Live subset (80.38% vs. 76.61%), which involves more realistic inputs. We attribute this to reduced reward hacking [25]: under fine-grained schemes, the model may overfit to superficial cues such as formatting or partial matches, without ensuring full execution correctness. Furthermore, within the binary setup, we observe that removing the reasoning format constraint significantly hurts performance (dropping from 80.38% to 76.24%). This highlights the critical role of structured reasoning in guiding Tool-N1-7B toward reliable and generalizable tool use, especially in complex, real-world scenarios.

Ablations on Training Data Composition. We then investigate how different data composition strategies affect performance on the BFCL benchmark. Experiments are conducted using the Tool-N1-7B model, with results presented in Table 4. Our key findings are as follows: (1) Compared to the raw model (Qwen2.5-7B-Instruct), R1-style training significantly enhances tool-calling capabilities. (2) ToolACE data yields particularly strong improvements in the live setting. Overall, these results suggest that progressively enriching the training data enhances the model’s tool-calling proficiency. (3) Compared to models trained using SFT on the same data, the R1-style training consistently yields better performance. Specifically, the Tool-N1-7B model trained solely on xLAM data outperforms the xLAM-8B SFT model by 6.36%, and the Tool-N1-7B model trained solely on the ToolACE subset exceeds the ToolACE-8B SFT model by 1.62%, despite using only a subset of the data.

6. Conclusion

We introduces Nemotron-Research-Tool-N1, a series of tool-using language models trained with a rule-based reinforcement learning. Unlike prior approaches that depend on supervised fine-tuning, Nemotron-Research-Tool-N1 leverages a reward function that supervises only the final answer and the structural format of reasoning. This allows the model to learn effective reasoning strategies without requiring annotated reasoning trajectories. Experimental results show that Nemotron-Research-Tool-N1 consistently outperforms existing baselines across multiple benchmarks, including BFCL and API-Bank. Furthermore, when trained on the same data, models using R1-style reinforcement learning achieve superior performance compared to their SFT-trained counterparts, affirming the benefits of reinforcement-based training over SFT.