Introduction

• LangGraph is built on top of LangChain and is completely interoperable with the LangChain ecosystem.
• One of the common patterns we see when people create more complex LLM applications is the introduction of cycles into the runtime.
• These cycles often use the LLM to reason about what to do next.
• A big unlock of LLMs is the ability to use them for these reasoning tasks. This can essentially be thought of as running an LLM in a for-loop. These types of systems are often called agents.

Example

• In a RAG application, a call to a retriever returns some documents.
• These documents are then passed to an LLM to generate a final answer.
• While this is often effective, it breaks down when the first retrieval step fails to return valid results.
• In this case, it's often ideal if the LLM can reason that the results returned from the retriever are poor and maybe issue a second (more refined) query to the retriever and use those results instead.
• Essentially, running an LLM in a loop helps create more flexible applications and thus can accomplish more vague use cases that may not be predefined.

Necessity

• You may want to always force an agent to call a particular tool first.
• You may want to have more control over how tools are called.
• Depending on the state it is in, you may want to have different prompts for the agent.

Key Terms

StateGraph

from langgraph.graph import StateGraph
from typing import TypedDict, List, Annotated
import Operator


class State(TypedDict):
    input: str
    all_actions: Annotated[List[str], operator.add]


graph = StateGraph(State)

• StateGraph is a class that represents the graph.
• You initialize this class by passing in a state definition. This state definition represents a central state object that is updated over time.
• Nodes in the graph update this state. Nodes return operations to attributes of this state (in the form of a key-value store). It will be more apparent when we discuss the Nodes section.
• The attributes of this state can be updated in two ways.
  1. First, an attribute could be overridden completely.
     1. This is useful if you want nodes to return the new value of an attribute.
  2. Second, an attribute could be updated by adding to its value.
     1. This is useful if an attribute is a list of actions taken (or something similar) and you want nodes to return new actions taken (and have those automatically added to the attribute).
• When creating the initial state definition, you specify whether an attribute should be overridden or added to. On the pseudocode above, it is the second type.

Node

• After creating a StateGraph, you then add nodes with graph.add_node(name, value) syntax.
• The name parameter should be a string to refer to the node when adding edges.
• The value parameter should be either a function or LCEL runnable that will be called. This function/LCEL should accept a dictionary in the same form as the State object as input, and output a dictionary with keys of the State object to update.

See an example in pseudocode below.

graph.add_node("model", model)
graph.add_node("tools", tool_executor)

There is also a special END node that is used to represent the end of the graph. It is essential that your cycles be able to end eventually!

from langgraph.graph import END

Edges

After adding nodes, you can then add edges to create the graph. There are a few types of edges.

The Starting Edge

• This edge connects the start of the graph to a particular node.
• This will ensure that the node is the first one to be called when input is given to the graph. The pseudocode for that is:

graph.set_entry_point("model")

Normal Edges

• These are edges where one node should ALWAYS be called after another.
• An example may be in the basic agent runtime, where we always want the model to be called after we call a tool.

graph.add_edge("tools", "model")

Conditional Edges

These are where a function (often powered by an LLM) is used to determine which node to go to first. To create this edge, you need to pass in three things:

1. The upstream node: the output of this node will be looked at to determine what to do next.
2. A function: this will be called to determine which node to call next. It should return a string.
3. A mapping: this mapping will be used to map the function output in (2) to another node.
   • The keys should be possible values that the function in (2) could return.
   • The values should be names of nodes to go to if that value is returned.

An example of this could be that after a model is called, we either exit the graph and return to the user, or we call a tool - depending on what a user decides! See an example in pseudocode below:

graph.add_conditional_edge(
    "model",
    should_continue,
    {
        "end": END,
        "continue": "tools"
    }
)

Compile

After we define our graph, we can compile it into a runnable! This simply takes the graph definition we've created so far and returns a runnable. This runnable exposes all the same methods as LangChain runnables (.invoke, .stream, .astream_log, etc) allowing it to be called in the same manner as a chain.

app = graph.compile()

Agent Executor

• We've recreated the canonical LangChain AgentExecutor with LangGraph. This will allow you to use existing LangChain agents, but allow you to more easily modify the internals of the AgentExecutor.
• The state of this graph by default contains concepts that should be familiar to you if you've used LangChain agents: input, chat_history, intermediate_steps (and agent_outcome to represent the most recent agent outcome)

from typing import TypedDict, Annotated, List, Union
from langchain_core.agents import AgentAction, AgentFinish
from langchain_core.messages import BaseMessage
import operator


class AgentState(TypedDict):
   input: str
   chat_history: list[BaseMessage]
   agent_outcome: Union[AgentAction, AgentFinish, None]
   intermediate_steps: Annotated[list[tuple[AgentAction, str]], operator.add]

See this notebook for how to get started

Chat Agent Executor

• One common trend we've seen is that more and more models are "chat" models which operate on a list of messages.
• These models are often the ones equipped with things like function calling, which make agent-like experiences much more feasible.
• When working with these types of models, it is often intuitive to represent the state of an agent as a list of messages.

As such, we've created an agent runtime that works with this state. The input is a list of messages, and nodes just simply add to this list of messages over time.

from typing import TypedDict, Annotated, Sequence
import operator
from langchain_core.messages import BaseMessage


class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], operator.add]

See this notebook for how to get started

Modifications

One of the big benefits of LangGraph is that it exposes the logic of AgentExecutor in a far more natural and modifiable way. We've provided a few examples of modifications that we've heard requests for:

Force Calling a Tool

For when you always want to make an agent call a tool first. For Agent Executor and Chat Agent Executor.

Human-in-the-loop

How to add a human-in-the-loop step before calling tools. For Agent Executor and Chat Agent Executor.

Managing Agent Steps

For adding custom logic on how to handle the intermediate steps an agent might take (useful for when there are a lot of steps). For Agent Executor and Chat Agent Executor.

Returning Output in a Specific Format

How to make the agent return output in a specific format using function calling. Only for Chat Agent Executor.

Dynamically Returning the Output of a Tool Directly

Sometimes you may want to return the output of a tool directly. We provide an easy way to do this with the return_direct parameter in LangChain. However, this makes it so that the output of a tool is ALWAYS returned directly. Sometimes, you may want to let the LLM choose whether to return the response directly or not. Only for Chat Agent Executor.

Future Work

We're incredibly excited about the possibility of LangGraph enabling more custom and powerful agent runtimes. Some of the things we are looking to implement in the near future:

• More advanced agent runtimes from academia (LLM Compiler, plan-and-solve, etc)
• Stateful tools (allowing tools to modify some state)
• More controlled human-in-the-loop workflows
• Multi-agent workflows

If any of these resonate with you, please feel free to add an example notebook in the LangGraph repo, or reach out to us at [email protected] for more involved collaboration!

Also Read

• LCEL