Exploring Autonomous LLM Agents for Capture The Flag problem solving using Prompt Engineering, RAG and Open Source LLMs

Capture The Flag (CTF) challenges are a great way to improve your offensive security skill set and test new tools in controlled environments. Only issue is once you’ve done the same stego challenge a million times, it can get boring and repetitive… What if we could leverage LLMs to help us solve these simple challenges and potentially highly attack paths, or even solve harder challenges?

A quick summary of CTF events: CTFs are basically hacking competitions where teams and/or individuals compete in a controlled environment where they are tasked with a number of challenges in different security categories and their goal is to solve the challenge and retrieve a “flag” (ie. a string that is usually easy to identify and usually follows a given format, eg. flag{this_challenge_was_easy}) . These event are a great way to test your skills and learn new things which can sometimes be applied to real world scenarios but will teach you to always try harder.

Recently, there’s been a few papers and articles coming out from different research teams and organisations highlighting the potential of using Large Language Models (LLMs) and LLM Agents in CTF events:

• Anthropic recently released their red.anthropic.com blog which highlights Claude’s capabilities in CTF competitions showcasing that LLMs can solve simple-medium CTF challenges, however they still struggle on harder challenges (eg. 0 challenges solved for Plaid CTF and DEF CON CTF Qualifier). They presented their research at DEF CON 33 which you can see here.
• A number of research studies have been published showcasing LLM’s abilities in solving CTF challenges as well as improvements when crafting specific Prompt and using LLM Agents to improve capabilities and solve harder challenges. As part of this blog, I’m going to focus on a specific paper which has achieved great results using Prompt Engineering; combining a number of prompting techniques to create more powerful Agents. You can read their research paper here and access their code here
• AIxCC a 2 year competition where teams were tasked to find and fix vulnerabilities in Open Source applications by leveraging Artificial Intelligence completed recently and every competing team had to release their code. This provided an insight into leveraging LLMs to solve cybersecurity issues at scale. Highly recommend spending the time into reading some of the articles of different teams. I suggest starting with Trail of Bits article on the topic.
• Earlier this month, wilgibbs released a blog about using open AI’s GPT-5 model to solve one of the challenges in the DEF CON CTF finals. DEF CON CTF is regarded as one of the hardest CTF competitions that runs every year and having an LLM solve one of the challenges highlighted the AI technological improvements that we’ve seen recently. You can read his blog post here.

That’s a lot of research to explore but after reading so many articles I wanted to also get my hands dirty and see what I could come up with. When I started looking into it, I wanted to set myself a number of goals:

• I wanted to focus on Open Source LLM agents and see how far they can be pushed to solve complex problems like CTF challenges. Closed Source LLMs are great but as LLMs providers become more like LLM Agents (I’ll explain what I mean with in the “What’s so special about GPT-5?” section below), we start wondering how much improvement is coming from the LLM itself vs Agentic improvements. Furthermore, for privacy reasons, companies might want to run their own LLMs instead of sending all their internal information and code to 3rd parties like Open AI.
• I wanted to attempt to use smaller but more specialised LLMs, for example using a 12B parameters LLM that focuses on only solving Cryptography challenges
• Ensure the Agent is fully autonomous such that the agent receives no user interaction (apart from starting the agent).
• Improve the capabilities of smaller agents by leveraging RAG and/or fine tuning

A quick introduction on LLM Agents

What are LLM Agents

An agent is an LLM-powered system designed to take actions and solve complex tasks autonomously. Unlike traditional LLMs, AI agents go beyond simple text generation. They are equipped with additional capabilities, including:

• Planning and reflection: AI agents can analyze a problem, break it down into steps, and adjust their approach based on new information.
• Tool access: They can interact with external tools and resources, such as databases, APIs, and software applications, to gather information and execute actions.
• Memory: AI agents can store and retrieve information, allowing them to learn from past experiences and make more informed decisions. ref: https://www.promptingguide.ai/agents/introduction

Agents represent systems that intelligently accomplish tasks, ranging from executing simple workflows to pursuing complex, open-ended objectives. OpenAI

One of the main selling points of Agents is that you can leverage them to perform more advanced tasks by giving them access to tools, guiding them to provide structured answers (eg. respond with the following JSON template) and chain tasks/sub-tasks until they solve your problem. CTF challenges can be quite complex and usually require multiple actions/tools in order to solve. Hence, they’re a great testcase for exploring LLM Agents and pushing its boundaries.

If you want to learn more about agents, I highly recommend looking at OpenAI’s guide on building agents .

What’s so special about GPT-5?

If you’ve played around with GPT-5 (especially its “Thinking” version), you might have realised how good it is at breaking tasks into sub-tasks until it solves your query. It also has access to tools which can be executed during any sub task. In the past, “thinking” traces appeared to be continuous and limited in their capabilities but with the release of GPT-5 the “Thinking” traces appear to be somewhat limitless, continuing until they solve your task. If you’ve played around with LLM agents before, you might have realised that this sounds very similar to some of the techniques used to augment LLM capabilities, namely:

• chain-of-thought process / ReAct Prompting
• embedded prompt-chaining

I explain some of these techniques in a later section: “Improving Agents using Context Engineering and Prompting Techniques”

Take for example, the following chat showing GPT-5 attempting a networking CTF Challenge from 247CTF.com. You can see it mapping out intermediate steps, trying out different tools and techniques before giving a final answer. The answer is wrong (hence why I’m showing it here) but the way it moves between sub-tasks is very interesting and very thoughtful, using python builtin libraries to parse the packet capture (pcap) file embed in the zip file:

After playing around and reading about GPT-5 a lot, I speculate that there are actually not many direct LLM improvements (ie. data/learning improvements) but instead they trained the model to be more agentic and improved the agent’s flow by leveraging recent prompting techniques (ie. Chain-of-Thought, ReAct, Prompt-Chaining), which I’ll explain in the next section.

With that in mind, this means that you may potentially be able to replicate or at least improve other models by incorporating them into similar agent systems. This is one of the reasons that I’m excited to try using Open Source LLMs (especially smaller ones) to see if they can be tuned to perform on par with GPT-5 and other large language models at specialised tasks like solving CTF challenges.

If you want to read more about GPT-5 specifically, I recommend the following articles:

• https://openai.com/index/introducing-gpt-5/
• https://botpress.com/blog/everything-you-should-know-about-gpt-5
• https://github.com/elder-plinius/CL4R1T4S/blob/main/OPENAI/ChatGPT5-08-07-2025.mkd
• https://fi-le.net/oss/

Improving Agents using Context Engineering and Prompting Techniques

A number of researchers have published techniques and prompting strategies that can be leveraged to enhance responses from AI agents. Using these enhancements allows LLMs to solve more complex challenges, more consistently and/or with fewer steps. They provide a great toolkit to builder who want to tackle more complex problems.

Context Engineering is a newer term which has started replacing prompt engineering as Agents have evolved to incorporate more than just better prompting. Its a more encompassing term which now includes things like Retrieval Augmented Generation (RAG) to add contextual information to chats, Structured Output to standardise the LLMs response into more predictable formats (ie. JSON) and keeping Memory of past events or actions to adapt LLM responses.

I’ll introduce a number of context engineering techniques which are discussed in this article and used by the Agent released alongside it.

Chain-of-Thought (CoT)

Chain-of-Thought sounds complicated but its actually very simple. The idea is basically to split a task into smaller (more manageable steps). To do this with LLMs, you basically ask them about the first step to solve a problem, then the second step and the next step, and so on until the problem is solved. Newer models who have “Thinking” abilities, can attempt to do Chain-of-Thought directly. However, its harder to manage then simply asking about one step at a time since the model is free to “think” and might get sidetracked quickly, hallucinate or end up in rabbit holes (error propagation).

As such, its easier to keep on querying the model for a single thought and then querying it again for the next thought while providing the previous thought. It allows us to manage the context we’re giving the model. We can provide less context such that it doesn’t overwhelm the model with information as research has shown that the more information (ie. context) you provide, the easier it is for the model to hallucinate, forget information and co.

Here’s a quick example of what Chain-of-Thought might look like in code:

from openai import OpenAI

client = OpenAI()

system_prompt = """
You are tasked with providing technical guidance, helping users install Arch Linux. 

Think step by step and only provide one step at a time. I will provide a list of steps we have taken so far, only give me the next step or command I need to do. 

If you believe we are done, only reply with "INSTALLATION COMPLETED!"
"""

done_trigger = "INSTALLATION COMPLETED!"

installation_succeeded = False

steps_performed = list()

# Maximum number of steps as fallback
# in case we don't succeed in installing Arch Linux
max_steps = 100

for _ in range(max_steps):
	query = ""
	
	# Add the step history to our query so the Model knows where we're at
	# and can think about the next step after this
	if steps_performed:
		query = "Here are the steps I have performed:\n"
		for j, step in enumerate(steps_performed):
			# Note: We could decided to truncate the step description 
			# as it might contain a lot of information and we only want a short summary
			query += f"step {j}: {step}\n"
	
	query += "What is my next step?"
	
	response = client.responses.create(
	    model="gpt-4o-2024-08-06",
	    input=[
	        {"role": "system", "content": system_prompt},
	        {"role": "user", "content": query}
	    ]
	)
		
	next_step = response.output_text.upper().trim()
	
	# The LLM thinks we're done
	if next_step == done_trigger:
		installation_succeeded = True
		break
		
	print(f"Here's our next step: {next_step}")
	
	# ... You could add code here to tell the model if you're running into issues with this steps
	# and provide some context such that the model tries to help you solve your issues 
	# before moving on to the next step
	
	# Adding step to steps performed 
	steps_performed.append(next_step)

print(f"Installation succeeded: {installation_succeeded}")

Planning

Planning simply involves asking the model to generate a plan instead of provide a description or single step. This does not seem super useful on its own since LLMs are trained to do this by default. However, when you are combining different agents for different tasks, it becomes a lot more interesting as you can use stronger models for planning, request agents to write plans for next few steps so they don’t get sidetracked and more.

Here’s a super basic example, which doesn’t add much value on its own:

from openai import OpenAI

client = OpenAI()

system_prompt = "You are tasked with providing technical guidance. When a user has a request, give the user a plan of all steps needed to solve his issue."

query = "I'm trying to install Arch Linux on this new computer, how do I do it?"

response = client.responses.create(
	model="gpt-4o-2024-08-06",
	input=[
		{"role": "system", "content": system_prompt},
		{"role": "user", "content": query}
	]
)
	
print(response.output_text)

ReAct

ReAct is a general approach that couples an LLM’s reasoning with concrete actions. By prompting the model to produce both intermediate reasoning traces and action steps, the system can plan, revise, and execute tasks dynamically, while also consulting external sources (eg. using web search tools) to bring in new information and refine its reasoning.

The best way to explain ReAct prompting is to look at the example provided in the research article which introduce the idea (Yao et al., 2022). For this example, they asked the following question: “What other devices, apart from the Apple Remote, can control the program originally intended for the Apple Remote?” and provided context to the model (ie. Thought, Action and Observation) which helped the model perform step by step analysis and enhance its response based on the information it retrieved from its actions (ie. tools):

If you want to read more on the topic I recommend reading the article itself or a quick rundown here.

Prompt Chaining

Prompt chaining refers to basically using different prompts to perform different actions. Our first prompt could be to request a plan from a model and then we create a second request that asks the same or another model to create an action from that plan (ie. combine our planning technique with another model that provides actions).

Splitting the prompts this way also helps with keeping context small, testing different models for different steps and limiting models to a single action per query instead of letting them go rogue (ie. easier to manage).

Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) is a method used to provide additional context to a model by search for information relevant to the task from a database/set of documents.

RAG takes an input and retrieves a set of relevant/supporting documents given a source (e.g., Wikipedia). The documents are concatenated as context with the original input prompt and fed to the text generator which produces the final output. This makes RAG adaptive for situations where facts could evolve over time. This is very useful as LLMs’s parametric knowledge is static. RAG allows language models to bypass retraining, enabling access to the latest information for generating reliable outputs via retrieval-based generation. ref: https://www.promptingguide.ai/techniques/rag

For CTFs, there are a lot of tools and terminal commands that you need to know about and a lot of the time you’ll have to research or look at examples to understand how to run the tools. If we’re able to provide these tools, use-cases and examples, the model can make use of those tools and might be more likely to solve the challenge. Provided that we give the correct tools and information in our query, which is easier said then done…

There are two main steps in RAG:

1. Retrieval: retrieve relevant information from a knowledge base
2. Generation: insert the relevant information to the prompt for the LLM to generate information

However, the retrieval part can be a little complicated and requires some preparation. You can’t simply provide everything you have to a model unless your knowledge base is small and/or the model’s context limit it high. To solve this, most RAG systems will store their database in text embeddings stored in a vector store database. Text embeddings are numerical vectors that represent pieces of text (words, sentences, docs) so that similar meanings have vectors that are close together. Vector stores are simply special databases optimised for vector similarity search (ie. find the vectors most similar to this other vector).

With this system, we can store our knowledge base in a special database as vectors and when we’re looking for something we simply convert what we’re looking for to a vector and search for similar vectors in the database.

Here’s a quick rundown of how this would work step by step:

# Before requesting information from a model 
1. Create a knowledge base
2. Split that knowledge based into documents (ie. chunks of text) and create text embeddings for each document
3. Store those embeddings in a vector store database
   
# When querying for information
1. Request the topic/question/query from the user
2. Convert that query to text embedding(s) as before
3. Search your database for the most similar vectors
4. Send the user's query to an LLM alongside the documents with the most similar vectors to help the LLM respond to the user's query
5. Retrieve the LLM's response and provide it to the user

This sounds easy but in practice, there are many edge cases which might affect the effective of your RAG Agent such as:

• Noisy, outdated, or duplicated documents leading to bad answers
• Chunking issues (eg. you have a knowledge base with bash commands but your chunks cut through those commands by mistake so you might have one part of a command in a chunk and one part in another)
• Using different embedding models
• Ambiguous queries (eg. “what’s the new policy?” - New relative to when? What type of policy?)
• Security issues like (indirect) prompt injection
• and a lot more…..

If you want to look at a basic RAG code example, I recommend looking at the following Jupyter notebook by mistral.ai.

Structured Output

JSON is one of the most widely used formats in the world for applications to exchange data. Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don’t need to worry about the model omitting a required key, or hallucinating an invalid enum value. ~ OpenAI

You basically give the model a format to follow, this will usually be a JSON schema. Some SDKs and inference providers support it directly so you don’t need to write pure JSON but instead can do something like this.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

# This is the format I want the model to follow
class MovieResponseFormat(BaseModel):
    title: str
    rating: int
    actors: list[str]

response = client.responses.parse(
    model="gpt-4o-2024-08-06",
    input=[
        {"role": "system", "content": "You're a movie title expert. Provie Movie recommendations based on the user's preferences."},
        {
            "role": "user",
            "content": "I like action and comedy movies",
        },
    ],
    text_format=MovieResponseFormat,
)

# The parsed value will be a MovieTitle object
movie = response.output_parsed

# And you can use it as such
print("Suggested movie", movie.title, movie.actors)

If you look at the API chat response, it should look something like this:

{
  "title": "My beautiful movie",
  "rating": 5,
  "actors": ["Jacob", "Adam", "Paul"]
}

Something to note, older or smaller models might not always follow response format instructions and therefore might respond with invalid json or no json at all, potentially causing bugs or exceptions in your code.

Making a simple interactive TUI Agent

Enough theory, lets get into the interesting stuff now and actually make an Agent and solve some CTF challenges.

My first iteration involved making a Terminal UI (TUI) Agent which supported tools that I could provide to the agent and chat with the agent directly and see the progress. I also wanted to have the ability to save each chat trace including tool calls and display it as a webpage so that I could see what tools were being called, the agent’s thought process, my user input, etc… You can view the code here (mostly vibe coded with codex).

To test the agent, I was using a CTF challenge from the awesome platform 247ctf.com. The platform has a variety of challenges and is a great way to learn about new and interesting hacking topics. One of the challenges I was able to solved using the Agent was a web challenge called Compare the pair:

This challenge involves bypassing a weak authentication check in order to retrieve a flag (I’m not going to spoil the solution in case someone wants to try it, otherwise you can just read the Agent’s chat trace below):

Here is the prompt I used which resulted in the Agent solving the challenge. Notice that I explicitly asked to not execute any actions and make a plan instead but it decided otherwise and was able to solve the challenge:

You are a ctf player, using the tools at your disposal solve the challenge. Request the challenge first and then make a plan of action. DO NOT EXECUTE any of the actions yet! When bruteforcing you can bruteforce up to the following number: 999999999. The challenge is: url: https://55affb983512588f.247ctf.com/ Can you identify a way to bypass our login logic? MD5 is supposed to be a one-way function right?. 

My first attempt was quick and dirty and ran commands in my virtual machine. It didn’t take long for models to start downloading and installing random tools… I learned my lesson quickly and move on to using a separate docker environment. You can find the docker contained MCP server I modified here.

It was great to test out various tools and play around with writing better prompts. However, it required a lot of babysitting and re-prompting. After playing with it for a while, I wanted to move on to something more automated that leveraged more advanced techniques and hopefully had a better chance at solving challenges. More specifically, I wanted to use context engineering techniques and see if I could an automated agent able to solve advanced problems more consistently.

Reviewing previous research

While researching the topic, I came across a multiple articles using benchmarks to test LLM and Agentic capabilities for solving CTF challenges. One of the newer articles on the topic comes from PalisadeResearch and showcased a comparison of different prompting techniques leveraged to solve a set of challenges from the picoCTF competition. picoCTF is an entry-level competition so the challenges themselves are not difficult but they teach a number of techniques that are required to know and solve harder challenges. Hence, getting a baseline on these challenges is a great way to ensure LLMs and Agents might be capable of solving harder challenges.

PalisadeResearch has made their code available which makes it easy to test out and build upon.

As my goal is to use open-source models, I wanted to get a baseline comparaison of their current capabilities using PalisadeResearch’s code before moving on. Only small modifications were made to allow using OpenRouter as an inference provider instead of OpenAI. I’ve uploaded the diff of changes here for transparency.

Note: I have not modified the code, prompts, environment or Agentic behaviour in order to keep a better baseline comparison. I understand that this code is made to compare different configurations but is not robust enough to be used in a production environment and does not handle agents misbehaving, not following response formats, etc. I’ve remade the application from scratch to better handle such cases which I’ll showcase in the next section.

I decided to use the following open-source models (logs can be found here):

* : Some models error’ed before completing the benchmark due to a number of issues (eg. not following response_format, running out of context space, inference provider issues, etc). Since my goal also involves making the Agent more robust, I am not going to try solving every error encountered yet and will use these tests as a baseline instead. I test a bunch of other models too but smaller models were very inconsistent with set response formats.

**: The first run of the ByteDance-Seed/Seed-OSS-36B-Instruct model failed due to provider issues (only 3 challenges were solved). However I wanted to get a better baseline as I decided to use this most for most of my testing later on. As such, I re-ran it a second time to get a more complete baseline.

***The two Qwen3-Next-80B-A3B models were released as I was about to release this blog. After testing them, they showed good results and seemed promising. The instruct (non-reasoning) model was also quite quick compared to some of the other models so I decided to test them out and use them in this research.

The fact that an open-source 80B parameters model matched a 685B model and was on par with strong closed source models is very promising. Furthermore, the ability of newer models such as DeepSeek-V3.1, Seed-OSS-36B and Qwen3-Next-80B-A3B to follow instructions and consistently provide accurate response formats has been quite impressive. Since I wanted to focus my research on smaller models, I decided to focus on one reasoning model (ByteDance-Seed/Seed-OSS-36B-Instruct) and one non-reasoning model (Qwen3-Next-80B-A3B).

This benchmark only compares solve rates, however since the newer models basically solve the same number of challenges, to be more thorough we should look at more than simply the number of solves. For example, solve speed and number of steps to solve could be taken into account to compare the different models. We’ll look at optimising these in the RAG section of this article.

Furthermore, these models have more training, better quality data and benchmark data contamination might be an issue so it’ll be important to test them against other challenges/benchmarks.

Developing a more robust Agentic solution to solve CTF challenges

After attempting to refactor the Palisade/intercode’s code twice (using GPT-5 codexand manually), I gave up and decided to rewrite everything from scratch. Too much of the code was useless or redundant (the original intercode research looked at more than just CTFs so more code was needed), prompts were stored everywhere and hard to manage. It was difficult to adapt new strategies and rework it, so instead I design an improved system and wrote it from scratch.

I will say that Palisade/intercode were only exploring different techniques for research purposes and had no intention of actually using the agents outside of benchmarks which is why they might put less time into designing a more robust and more versatile system that can be used for benchmarks as well as standalone tasks (ie. used during CTF events).

Palisade’s research showed improved solve rates using prompting techniques and agent loops such as the ones we discussed earlier (eg. Chain-of-Thought, Plan, React, React-Plan). They had developed the following 5 strategies:

• N-turns: this strategy is basically using Chain-of-Thought. The model is given the initial task and asked to perform 1 step each time for N number of turns. Each turn the agent is provided with the previous actions (ie. bash or python commands) performed and the action output (ie. stdout/stderr). The agent continues until it runs out of available turns or it solves the challenge. We’ll call this agent the ActionAgent.
• Plan: this strategy is similar except that there is a secondary agent which provides a plan on how to solve the challenge. This plan is given to the ActionAgent in an attempt to help it solve the task. We’ll call this secondary agent, the PlanningAgent.
• ReAct: this strategy involves two (2) steps. The first step is to ask an agent (ie. ThinkingAgent) to generate a thought about how to solve the next step, forcing the agent to think about reasoning steps before converting its reasoning to an action using the ActionAgent.
• ReAct+plan: this is basically the same as the ReAct strategy except also request a plan from the PlanningAgent and give that plan to the ThinkingAgent. This is the strategy that yielded the best results in their testing.
• Tree-of-Thoughts (ToT): this last strategy uses Tree-of-Thoughts which is another prompting strategy which I will explore in a future blogpost.

From the strategies above you might have spotted a common theme, basically the ActionAgent, ThinkingAgent and PlanningAgent can be combined and interchanged to make up 4 of the 5 strategies. Combining them as such makes the code (and strategies) more modular as you can interchange them, and/or add new agents with new abilities. For example, a Vision Agent to extract information from images and videos or a DeepSearch agent to search for information and resources on a specific topic (or PoC/CVEs) when we’re stuck on a challenge. This is why I decided to refactor them as such. It also makes it easier to manage the system and user prompts as they’re basically always the same regardless of the strategy.

With the modifications added and the cleaned up prompts you can see each agent here:

• ActionAgent
• PlanningAgent
• ThinkingAgent

The ThinkingAgent seems quite redundant for reasoning enabled models as they already have their own thinking. It might be more helpful for non-reasoning agents as it provides them the ability to reason before giving an action command. It would be interesting to test different setups in the future with and without the ThinkingAgent thoughts as I speculate that it might not provide much value except giving the agent an additional step/attempt to solve challenges. With the improved response format, the models are also asked to provide an explanation of their action which basically describes the thought behind their command and I believe could replace the “thought” from the thinking agent:

I have kept the ThinkingAgent for now as I test out different setups (and since I wanted to be able to replicate the previous study) and check if this step is needed or not. I also believe that with finetuning, a single agent setup might be able to replace the PlanningAgent and ActionAgent setup with a single agent that does everything and chooses when to make a plan vs tries to run a command.

In addition to separating each agent, I’ve also added a response format for each agent which provides structured responses and allows better context management when building the query (ie. when providing the thoughts or plan to the action model). Here’s a before and after adding a response format to the PlanningAgent, you can see the unstructured nature of the first response which would have been simply appended to our query and might cause confusion while the second version is more structured and allows us to test different version by providing more or less information to the ActionAgent (eg. providing suggested tools):

Running benchmarks is nice but all you get is number go up… What if we actually want to use our Agent during a CTF? For that reason, I added a Jupyter notebook with boilerplate code to download and setup a new Task (aka a challenge) and give it to the Agent to solve. Here’s an example where I’m using it to solve a networking challenge from 247ctf.com:

It took its sweet time but got there in the end… There’s still lots of improvements possible, especially around providing help with how to use certain tools but it was stubborn enough and managed to complete it! There’s a few ideas we could leverage here like attempting to learn from our solve by looking at the shortest solve path based on the commands we ran and saving those commands for later use. We’ll see how we can incorporate some of that in the RAG section below.

In addition to the changes above, here’s a non-exhaustive list of additional features and improvements made:

• Code refactored for modularity, removing all unused code and combining as much as possible

• Environment upgrades

  • Added missing tools and created symlinks to help with models attempting to use tools in different manners (eg. RsaCTFTool.py instead of RsaCTFTool)
  • Each new run is performed in its own container environment meaning you can do multiple instances at once in different terminals
  • The challenges are copied during the task itself and not during container creation which makes it easier to test out new challenges and cleanup environments
  • Improved container deletion once benchmark/task has completed.

• Usability & Reliability improvements

  • Added a Jupyter notebook with a single task solver to use during CTFs
  • Added a shell script to run OpenRouter agents
  • Added a number of checks and conditional retries with exponential back-off to prevent issues with inference servers
  • Fixed a number of bugs, errors and added soft-fails where possible
  • Added multi-threading for benchmarks so you don’t have to wait hours for it to be completed

• Observability and Operations

  • Added the ability to setup Langfuse to help with LLM Observability (see Observability and Operations section for more information)

• Agents

  • Cleaned up agent prompts and added context to improve task solve speed, solve rate and solve consistency
  • Removed prompt sections like “I’ll tip you 100 dollars”. While this worked great some time ago, this is usually no longer required.
  • Centralised Prompts
  • Added additional context such as flag format, provided files, URLs and challenge category.
  • Added ResponseFormats to Planning and Thinking agents who didn’t have a specific response format
  • Added RAG (See Retrieval Augmented Generation (RAG) for more details)
  • Added arguments to vary the planning strategy based on either static number of steps or every X step.

• Logging

  • Cleaned up the log file naming convention to prevent overwriting runs
  • Added better logging overall and storing more information like start-time, end-time for tasks, runs, etc

Missing features from previous research:

• Currently does not support ToT strategy, however I will add this in the future.

For a list of planned improvements, see the Future research and tool improvements Section.

Benchmark comparison post-rewrite

To ensure I hadn’t created a worse setup, I decided to run the benchmark again on models tested previously. The results are on-par with Palisade’s research except with improvements on reliability (ie. gpt-oss-120b jumped from 9 to 75 challenges solved).

The updated table can be seen below. Its not cheap to run these and takes a while so I only re-ran a few of them which would give a good idea where we stand:

*The large negative deltas for Qwen/Qwen3-Next-80B-A3B-Instructare due to inference provider errors and internet connection cuts. These benchmarks are expensive to run so I’m not going to run these again and will assume that they are negligible for now. I also only did 1 pass through and 2 attempts for each task instead of the usual 3 attempts per task.

The -1 delta discrepancy for models DeepSeek and Seed-OSS models is due to less steps allowed (ie. 25 instead of 30) and less attempts (ie. 2 instead of 3) than the previous benchmark. Hence, I’ll consider these on-par. The biggest improvement can be seen with models that previously had very low score in the baseline due to a number of issues (ie. response formatting, poor error handling, bugs, etc). This shows that we’ve managed to improve consistency and robustness quite a bit, allowing smaller models such as Mistral Nemo to perform significantly better while only being 12B tokens in size.

Retrieval Augmented Generation (RAG)

There exists a tool called RsaCtfToolwhich basically tries a number of insecure RSA primitives to recover a private key and use that private key to decrypt encrypted text (ie. ciphertext). In their research, Palisade modified the prompt to mention that we have access to the tool, installed the tool in the environment and added it next to the challenges which might require it :

They also provide an example in one of their prompts:

This tells the model the tool exists, helps it learn about the tool and how to use it. While this is not being retrieved dynamically, if we were able to retrieve tools based on the current task, we could list a lot more tools that might help the agent solve the task. This is where RAG comes in. RAG enables us to retrieve tools and information at runtime based on current task and the theories we have to solve the challenge.

Lets look at examples of how that can improve our agents.

Enhancing capabilities by introducing tools: RsaCTFTool for crypto challenges

One of the challenges in the picoCTF benchmark can be solved entirely using RsaCTFTool. The problem is if the models are not aware the tool exists or don’t know how to use the tool properly they will try to solve the tool manually which may work but is very inefficient.

The challenge is a crypto challenge that tries to teach you about RSA implementation weakness, specifically using non-prime number for N which can be factorised relatively easily:

The quickest way to solve this challenge is to use RsaCtfTool to search a online database of factors (ie. factordb.com) and factorise N. This allows us to recover the private key:

RsaCtfTool also has a way to decrypt ciphertext directly which mean with one command you can basically solve the challenge:

$ RsaCtfTool -n 1422450808944701344261903748621562998784243662042303391362692043823716783771691667 -e 65537 --decrypt 843044897663847841476319711639772861390329326681532977209935413827620909782846667 --attack factordb --private
['/tmp/tmpkbo3txdj']

[*] Testing key /tmp/tmpkbo3txdj.
[*] Performing factordb attack on /tmp/tmpkbo3txdj.
[*] Attack success with factordb method !

Results for /tmp/tmpkbo3txdj:

Private key :
-----BEGIN RSA PRIVATE KEY-----
MIGwAgEAAiIv/IZi84fX6oy8X46Nkz8/hpvqZIr+AVHktXPVdOmciSqTAgMBAAEC
IiDlTT8CUaqryOTN4Qye16nwq1dq9EdZ7q2VP9DPJWALAaECEQZY9sOHJWE7+0FH
2V+wZK0DAhIHj1QAU1FI0d/1sfIrHodVrzECEQP5H8zYBT/NycHc9WV+BiZ9AhE9
KENmXqA1eaVS/TsdmNs9UQIRAPFen0CtiPHZkuODgHqHqRA=
-----END RSA PRIVATE KEY-----

Decrypted data :
HEX : 0x007069636f4354467b736d6131315f4e5f6e305f67306f645f30303236343537307d
INT (big endian) : 13016382529449106065927291425342535437996222135352905256639555294957886055592061
INT (little endian) : 3710929847087427876431838308943291274263296323136963202115989746100135819907526656
utf-8 : picoCTF{sma11_N_n0_g0od_00264570}
utf-16 : 瀀捩䍯䙔獻慭ㄱ也湟弰で摯た㈰㐶㜵細
STR : b'\x00picoCTF{sma11_N_n0_g0od_00264570}'

If we look at how the model attempts to solve the task when it doesn’t know about RsaCtfTool, we can see it understands what it needs to do but tries to do it manually and factorise locally which could take a while, if even possible on a laptop. You can see the PlanningAgent’s response where its trying to use python to do it manually:

And the action models does as suggested using python:

We’ll now teach the Agent about the tool itself by adding the tool, its utility and some examples in the system prompt as such:

After providing tools and example commands, the agent has learned about the tool and how to use it and is able to solve the challenge within the least amount of steps (we can’t solve any faster then this). Here’s the plan returned after the tool information was added to the system prompt which now contains references to RsaCtfTool:

And the list of actions is now very direct, the ActionAgent also realised that it could do step 2 and 3 of the plan at the same time using 1 command and went for it:

Hence we can teach agents to use new tools available to them.

Reducing step count by giving command examples: recursive binwalk

Introducing new tools is great but we can also leverage this new ability to teach the model about how to use effective tools efficiently. Here you can see the model attempting a different challenge where it has to run a set of commands recursively until it gets to the flag:

The model is not aware that binwalk the first tool it uses has a way to run recursively which we can try to teach it:

After updating our system prompt we can see that the agent is now much more efficient, although its still not that great at navigating directories recursively:

By adding an example to our tool, we can teach it to be more efficient after extracting files:

With this new example, it decided to use the first example but then took ideas from the second example to find all files instead of recursively looking inside of each directory:

Why are there two cat commands? This is the output from the first cat command: p�i�c�o�C�T�F�{�4�f�1�1�0�4�8�e�8�3�f�f�c�7�d�3�4�2�a�1�5�b�d�2�3�0�9�b�4�7�d�e�}

Explanation from the LLM itself: The observation from cat flag.txt shows the flag characters separated by null bytes (visible as blank boxes). To get the clean flag, we need to remove these null bytes using tr -d '\0'

Adding RAG to our Agent system

We’ve just shown that its possible to improve capabilities by adding tools and examples to system prompts. Since my current list of tools is small I could technically just extend the system or user prompt with the tool list and examples. However, as the list of tools expands this won’t be manageable.

As such, we need to be able to only provide the tools needed for the task at hand. This is where RAG comes in!

The hard part with RAG is knowing what data to add to your context and when to add it. For this solution, I’ve decided to use multiple steps to figure out which commands and tools might be useful for the task. These tools and commands are stored in a database which are essentially json files. Here’s an example for the binwalk tool which has some lesser known arguments like -M to extract recursively:

{
    "title": "binwalk",
    "explanation": "Binwalk can identify and extract files and data that have been embedded inside of other files. Its primary focus is firmware analysis, it supports a wide variety of file and data types. Through entropy analysis, it can even help to identify unknown compression or encryption.",
    "examples": [
        {
            "query": "Recursively scan extracted (-e) files and data like matryoshka dolls (-M)",
            "command":"binwalk -eM <file.ext>"
        },
        {
            "query": "Recursively scan extracted (-e) files and data like matryoshka dolls (-M) plus print all extracted files",
            "command": "binwalk -e -M dolls.jpg && find . -type f"
        }
    ],
    "categories": [
        "reversing",
        "forensic"
    ],
    "tags": [
        "reversing",
        "forensic"
    ]
}

First, the planning agent receives a list of tools and descriptions based on the task’s categories. Using these tools, it generates a plan and might include the tools in their tool suggestions. There are no examples, just a list of tools that match the challenge’s category:

In its plan, the agent returns a list of suggested tools which might include some of the tools we retrieved from our database:

Now we want to convert the planning agent’s output to search for related tools and get relevant examples in order to provide this information to the thinking/action agents. We do that by using a hybrid RAG setup. We find the most relevant tools to add by calculating a score using a number of methods:

• BM25: basically a ranking algorithm that determines relevance based on keyword matching and word frequency in documents (ie. keyword matching)
• Semantic Search: Semantic search is all about understanding meaning and context instead of just word matching. This type of search focuses on understanding the intent behind the words in a query. Basically, we use an LLM to try and extract meaning from a query so that a query like “my laptop dies overnight while sleeping” does not end up with the agent trying to call the cops on you but instead find something like “Troubleshooting standby power drain and unexpected battery loss during sleep mode.”.
• Keyword Bonus: We have the luxury of having specific keywords that are more important than others. I’m talking about the tools/commands specifically (eg. binwalk). This means we can use it to boost entries that are about this tool specifically. This prevents the tool being buried if the query ran matches other commands more than this tool itself.

We generate a score for both BM25 and Semantic Search which we combine (using Reciprocal Rank Fusion) and then add the Keyword Bonus, mathematically this looks something like this:

score = RRF(BM25(query),SS(query)) + KB(suggested_tools, card_tools)

where
	query -> task_summary provided by the planning agent
	suggested_tools -> a list of suggested_tools provided by the planning agent
	card_tools -> a list of cards and associated tools for each card
	
	RRF -> Reciprocal Rank Fusion
	BM25 -> lexical search using Best Matching (ie. BM25)
	SS -> Semantic Search
	KB -> Keyword Bonus

Using this score, we rank the documents (ie. tool cards) and use a reranker which is a type of model that given a query and a set of documents will output a similarity score. We use this similarity score to reorder the tool cards by relevance to our query and finally we remove all cards below a minimum score as they are probably not relevant (used a min score of 0 while testing). This aims to provide the most relevant tools for the task at hand and hopefully helps us solve harder challenges (ie.RsaCTFTool example) or solve challenges faster/more efficiently (ie. binwalk example).

To show this works, we can attempt challenges 12 and 14 again (ie. the binwalk and rsactftool challenges) from before and see the improvement with and without RAG.

Without RAG, its trying python shenanigans hoping it can factorise it on this limited docker container inside of my limited VM on this old laptop which it’ll struggle with and timeouts before anything interesting happens. We can see the model does have limited knowledge of rsactftool and attempts to use it but fails miserably because it doesn’t know how to use it properly:

Similarly for the “binwalk” challenge, it uses the slow method of extraction, repeatedly calling binwalk, unzip, and dd:

With RAG, it solves both challenges in the smallest amount of possible steps using the tools exactly as we’ve provided them with in our RAG solution:

You can find both log traces here:

• With RAG
• Without RAG

This shows we can successfully leverage RAG to augment our agents and solve challenges that it might not have solved on its own.

Limitations of RAG

RAG is great when it provides the context required for a certain task but finding the right context (in our case the right tool(s)) and ensuring that we don’t poison the context with garbage data that is not useful for the task at hand is much more difficult.

There are also a number of limitations that have possible solutions which I will explore in the future:

• The context window is not unlimited: At the moment, its easy because we have a low amount of tools but when more tools are added, we might need more tools and tool examples and the context grow quickly
• Our current setup relies on the Planning Agent’s query which might not be accurate and might not match the correct tool(s). For example, challenge descriptions can be cryptic so when the planning agent attempts to solve them at first, it might not have enough information about the task at hand and end up suggesting completely incorrect tools. An idea could be to try and retrieve the tool cards later on and not just for the planning agent.
• It’s hard to keep the wiki up to date: At the moment, the RAG wiki only has 4 tool cards, namely binwalk, mergecap, rsactftool, ssldump. However, there’s a lot more tools that would be great to add in there but its hard to manage manually. What would be interesting is to use past/future solves and extract the set of commands required to solve the challenge and add them to our wiki automatically.
• When to add RAG context: The RAG context is only added when the planning agent is called (which might not always be the case). I need to find a better solution on when to add context vs when not to.
• Ambiguous tools: Some tools might be universal and not just relevant to crypto challenges for example and therefore would not be shown to the planning agent in our current setup. We might need to rethink how we list the tools relevant for the task when querying the planning agent in order to pull tools from categories other than the challenge’s category. For example, we could try a similar thing as we do with the other agents and use the category as a bonus keyword when we’re querying the cards for the planning agent.

Lastly, we could look into finetuning to embed the information directly into an LLM.

RAG vs Finetuning

RAG is when you want to bring to the LLM specific information. Fine tuning is when you want that knowledge set to be part of the system. ~ Xtianus21 (reddit)

RAG can help with giving knowledge of tools and concepts to LLMs by providing it within prompts and queries. The problem is it won’t improve the model’s task prioritisation and execution flow. For example, here the model knows about a few steps used to solve the task but instead of running one command at a time and checking it executed as expected its trying to perform all steps at once which is very error prone:

During testing, certain models had issues with tshark command and its various flags, filters, etc so it had to run similar commands quite a few times. It also failed to understand that it would be easier to run mergecap first and only once instead of running it on every command:

I will explore finetuning in a future article to see if it can help with solving these issues.

Testing improvements on a new benchmark

Why do we even need a new benchmark?

Large Language Model (LLM) benchmarks provide consistent, reproducible ways to assess and rank how well different LLMs handle specific tasks. They allow for an “apples-to-apples” comparison—like grading all students in a class on the same tests. Limitations of LLM benchmarks include potential data contamination, where models are trained on the same data they’re later tested on, narrow focus, and loss of relevance over time as model capabilities surpass benchmarks.

Benchmarks are nice but as LLMs get better and are trained with more data, a lot of benchmark data is also fed into LLMs which allows them to solve that specific benchmark but their abilities might not transfer to other benchmarks. It is said that some models are also directly trained on benchmark data in order to achieve better results for said benchmarks. This is called benchmaxing.

picoCTF is very well know and there is a plethora of write-ups available online. This means that the updated models Palisade have used (ie. OpenAI models) are likely to have direct solves for each of the challenges in their training data.

Furthermore, one of other the issues I have with Palisade’s research is that for Crypto RSA challenges, they explicitly put the RsaCTFTool inside the challenge folder as can be seen on challenge 12 and challenge 79:

This is basically a huge hint for the challenge itself. The model will most likely run ls -al to list the files in the directory and will see that there is the challenge as well as the RsaCTFTool in there and therefore attempt to use the tool to solve the challenge. The original research did not provide the tool with the challenge itself. I feel like that is a big hint to give to the LLM as CTF challenges usually don’t include tools with challenges but instead require the user to identify fitting tools for the challenge.

They also mention that the tool is available within one of their system prompts although that is considered context engineering which in my opinion is fine:

Nevertheless, since there is only one tool listed it seems very “optimised” for the benchmark itself. Additional tools or a RAG integration would have been more impressive.

Regardless, I’m not here to trash on their research, I just want to highlight some of them reasons why I believe we need more benchmarks.

Creating a new benchmarks

Creating a new CTF benchmark is quite simple considering the amount of open source challenges and CTF archives on Github, ctftime.org, and the web.

As a test run, I decided to create simple benchmark using a total of 27 CTF challenges in varying difficulty. A lot of these challenges are quite a bit harder then the picoCTF challenges, however I know for a fact that a number of these can be solved with LLMs since I was able to solve a few using GPT-5 Thinking directly within OpenAI’s chat interface (ie. no access to Kali environment).

The benchmark, which I’ve named The Unfinished CTF Benchmark, is comprised of the following 27 challenges of varying difficulty, split in five different categories:

Note: I’ve decided not to release the benchmark as some of the challenges may still be active and I’m still adding new challenges to it. I may or may not release a curated benchmark in the future… Only time will tell!

Comparing Agents on the new benchmark

// TBD - comparison in progress

Observability and Operations

LLM observability is the practice of gaining comprehensive, real-time visibility into the behavior, performance, and output characteristics of large language models (LLMs) and their associated applications in production. It goes beyond simple monitoring by providing the ability to understand the internal states of an LLM system through its outputs, enabling teams to debug issues, optimize performance, and ensure reliability, safety, and efficiency. This is achieved by collecting and correlating telemetry data such as logs, metrics, and traces from the application, APIs, and workflows.

Observability tools allow you to see chat traces without having to print them in logs or terminal outputs and provide a platform to improve Agents through dataset benchmarking/evaluation, prompt management, and more. For my use case, I only wanted to be able to log LLM chats, search them easily and identify any shortcomings or improvements that could be made to the CTF Agent I created. As such I decided on using Langfuse, an open source observability tool and one of the many available out there. The thing that sold me was the clean UI, ability to self-host and the simplicity with which you can integrate it into your applications to start logging chat traces.

In your Python application, replace the openai import with the langfuse equivalent

# import openai
from langfuse.openai import openai

Export the required environment variables and simply run your python agent:

export LANGFUSE_SECRET_KEY=sk-lf-....-2300a4e4dea2
export LANGFUSE_PUBLIC_KEY=pk-lf-....-dd2171f8d0a7
export LANGFUSE_HOST="http://localhost:3000"

python3 agent.py

That’s it… That’s all you need to get basic traces logged in Langfuse! I mostly used the Observability feature which provided LLM chat/session tracing so I can identify any issues or improvements. I will explore other features in the future. You can see some examples below:

It’s a great way to see if something is wrong, for example here I’m failing to provide a turn history:

Testing improvements in a live CTF

Before I test the Agent in an actual live CTF environment, I needed it to have a hacker alias, and decided to let our LLM overloards choose. Here’s what GPT-5 came up with:

I also asked it for a Country and Team Name as some CTFs require that information:

Username: rop_n_roll
Country: Estonia
Team Name: Baltic Bitflip

So if you see a player named rop_n_roll from Estonia part of the Baltic Bitflip Team competing in your CTF event, you might be playing against a bot.

Unfortunately, I only had time to test a single challenge due to IRL commitments, however rop_n_roll battled through and was able to solve that it:

Great success!

Conclusion and research outcomes

Research Outcome

Learned a lot through this, understanding how agents and models work, how to improve their capabilities what they are good at and what they might need help with… The Agent I made is very early into development, there’s still a lot of improvements possible which I will definitely explore in the future. Although future articles will be more to the point… Nevertheless, I was able to augment capabilities using RAG which are opening up ideas for future enhancements and hopefully help small open source LLMs bridge the gap and compete with large closed source models.

The agentic solution has already proven itself against live events! I will definitely do more testing in future events and try to see what can be improved further.

Furthermore, I’m excited to try and incorporate self-improvement into RAG by automatically saving and optimising solve paths into command examples and what not as well as trying out finetuning. Obviously more data will be needed for this but with the new setup, I believe it will be easier to create new benchmarks from past and future CTFs and save challenges to train on. Also, I believe adding vision capabilities and deep research could be interesting and might be required to solve certain challenges. Food for thought.

The future of CTFs

Are CTFs doomed? Short answer, probably not… CTFs have always been a good way to gain knowledge in new topics and a still an amazing way to learn and hone your skills in training environments.

If you play CTFs to learn about cyber security, this will not change and will still be a great way to learn. There are and still will be CTFs aimed at newcomers and people wanting to explore new skills.

If you’re playing CTFs competitively, you’re going to have to adapt quite a bit. LLMs are here to stay and are now able to do a lot of the grunt work. You’ll need to learn how to leverage them to gain an advantage against other teams. Historically difficult CTFs will likely adapt and become more and more challenging as you’re now expected to leverage LLMs to help solve challenges. Exploit path will likely be longer or more convoluted.

While everyone adapts to this change, I’m sure a lot of CTFs are going to experience unexpectedly large number of challenge solves as people leverage LLMs more and more. It’ll make for great drama on twitter with people confusing it for flag sharing and what not:

Note: I played in this CTF and know for a fact that GPT-5 Thinking was one-shotting a lot of these…

Future research and tool improvements

Performing this research has given me a number of ideas on what to improve in future iterations. I’ve added a starter list below which I hope to explore in follow up research:

• Finetuning models
  • Data generation
    • Synthetic dataset using OpenAI GPT-5 (or equivalent model) and a testing environment that validates the outputs and tools used
    • Using solves but removing intermediate steps which failed to identify optimised solves
  • Finetuning for agentic use
    • RL
      • https://github.com/OpenPipe/ART
      • https://www.youtube.com/watch?v=gEDl9C8s_-4
    • Finetuning a single agent instead of 3 separate agents - https://arxiv.org/abs/2509.06283
  • Tool use
    • Finetuning for tool use, python or bash commands
    • Fine tuning on kali linux commands
    • Fine tuning on better code search tools like ast-grep and other tools like strangerstrings
  • Trial fine-tuning using Lora, Qlora, etc…
  • Finetuning a reasoning model vs direct response model
• Agents / Context engineering
  • Adding a Deep Research Agent to search for tools, CVEs, topic knowledge and what not which may help provide more context and lead the agent to solving the challenge
  • Add an image parsing tool/agent to help describe, extract data or solve challenges which require looking at an image (ie. moondream 2B vision model, GLM-4.5V, etc)
  • Text recognition using https://github.com/open-mmlab/mmocr?tab=readme-ov-file , https://github.com/PaddlePaddle/PaddleOCR or tesseract
  • Add new tools like IDA/Ghidra MCP for reversing and pwn challenges and burpsuite or equivalent for Web challenges
  • Using specialised coding agents to analyse code for example during web or crypto challenges (ie. qwen coder, etc)
  • Test out and explore direct tool use instead of single step querying as is currently done
  • Explore specialised agents per task type
  • RAG
    • adding synthetic RAG, providing RAG based on the step (planning, thought, action)
    • self-learning mode - automatic RAG addition based on tools/commands that work well
    • test out other RAG solutions like weaviate or postgres vector
    • hybrid RAG retrieval
      • https://www.sciencedirect.com/science/article/pii/S1474034625001053
      • Using Weaviate: https://ragaboutit.com/how-to-build-a-hybrid-rag-system-with-weaviates-new-multi-vector-search-the-complete-implementation-guide/
  • Prompt engineering
    • Test out prompt optimisation manually and using frameworks like dspy
    • Test different system templates - not all models respond to same message templates, adjust for each
• Benchmarks / Training data
  • Adding an Improved Jupyter notebook to allow periodic saving of Task and Benchmarks for future comparisons
  • Add more challenges to train and benchmark against
  • keep all traces and use them for learning / improving models
    • remove non-working steps (ie. command not found, etc)
    • use traces to improve RAG or teach new models
• Other
  • Using a less bloated environment (ie. Kali is too bloated)
  • Adding a web UI to view output and with the ability to talk to the agent(s) and provide additional context and ideas
  • Testing with locally ran and parameters optimised by models
  • LLM Cache (eg. https://github.com/LMCache/LMCache) - Not really useful for personal use agents but interested in trying it out

Appendix

Using Vision Language Models to help solve CTF challenges

I’ve been thinking about using image models to help solve certain challenges like the picoCTF challenge below (although Vision Language Models might not be necessary here, you could probably do better with OpenCV). I first tried with the Moondream.ai vision model which is a 2B parameters model but very capable from previous testing, its not bad but needs some work:

I managed to get better results using GLM-4.5V model which correctly extracted the text from the image:

I also tested it on another challenge which uses NATO flag signaling although both models didn’t seem to understand it. More testing to be done…

Topping the leaderboards

Winner winner chicken dinner:

The only provider decided to stop providing the model (feelsbadman):

Plain LLM Agents

Plain LLM agents are simple agents that use a large language model (LLM) directly to choose next actions or generate task steps without extra layers like complex planners, learned policies, or specialized orchestration frameworks.

Key characteristics:

• Single-step decisioning: the LLM is prompted to decide the next action each turn (e.g., call a tool, ask a question, produce text).
• Minimal state management: little or no explicit memory, belief model, or long-term planning beyond what’s kept in the prompt/history.
• No learned controller: decisions rely on prompt engineering and the LLM’s reasoning, not on a separate trained policy network.
• Tool-driven behavior: often constrained to a fixed set of tools or API calls the LLM can invoke via structured outputs.
• Reactive and iterative: acts, observes results, and prompts the LLM again—adapting only through updated context.

When to use:

• Prototyping agents quickly.
• Tasks where short-horizon, conversational reasoning suffices.
• Systems prioritizing simplicity and interoperability.

Limitations:

• Poor scalability for long, complex plans.
• Fragile to prompt drift and verbose histories.
• Limited ability to optimize across multiple steps or maintain consistent long-term strategies.

Additional References

Prompt engineering:

• https://www.promptingguide.ai/
• https://google.github.io/adk-docs/agents/workflow-agents/loop-agents/
• https://github.com/elder-plinius/CL4R1T4S/blob/main/OPENAI/ChatGPT5-08-07-2025.mkd

CTF Agents research papers and blog posts:

• https://arxiv.org/abs/2412.02776
• https://arxiv.org/pdf/2403.05530
• https://arxiv.org/pdf/2403.13793
• https://enigma-agent.com/
• https://wilgibbs.com/blog/defcon-finals-mcp/
• https://github.com/aliasrobotics/cai
• https://github.com/enigma-agent/
• https://github.com/princeton-nlp/intercode/

Limitations of RAG: - https://www.reddit.com/r/OpenAI/comments/1bjtz7y/when_do_we_use_llm_fine_tuning_vs_llm_rag/ - https://docs.unsloth.ai/get-started/beginner-start-here/faq-+-is-fine-tuning-right-for-me#is-rag-always-better-than-fine-tuning - https://x.com/rohanpaul_ai/status/1961990185698337156 - https://arxiv.org/abs/2508.21038

Open Source LLMs used:

• deepseek v3
• llama 3.3 70b
• llama 7b/8b
• mistral nemo 12b
• gpt oss 120b
• bytedance seed 36b