Blog

Glossary · 82 terms

AI Agents & AI Coding Tools Glossary

Plain-language definitions of the terms used in AI agents and AI-assisted software development, each linked to the primary-source statistics that quantify it.

AI Agent Glossary

41 terms · updated 2026-09-13 · companion to AI Agent Statistics 2026.

  1. Agent host (runtime) The application that runs the agent loop, manages context, executes tools and enforces limits.
  2. Agent loop The observe → think → act cycle an agent repeats until it stops.
  3. Agent memory Storage an agent reads and writes across steps or sessions, outside the context window.
  4. Agentic AI AI systems designed to pursue goals with limited supervision by planning, acting and adapting.
  5. Agentic coding Delegating whole programming tasks to a coding agent rather than accepting line-by-line suggestions.
  6. Agentic workflow A multi-step LLM pipeline whose steps are fixed in code, with the model filling in each step.
  7. AI agent A system in which an LLM decides which actions to take, observes the results, and repeats until a goal is met.
  8. Autonomy level How much of the decision-making an agent performs without a human approving it.
  9. Benchmark saturation When top models approach a benchmark's ceiling, so it stops discriminating between them.
  10. Browser agent An agent that navigates and acts on websites through a controlled browser.
  11. Chain-of-thought (CoT) Having the model write intermediate reasoning steps before its answer or action.
  12. Coding agent An agent that reads, edits, runs and tests code in a repository to resolve a task.
  13. Computer use An agent operating a real GUI — screenshots in, mouse and keyboard actions out.
  14. Context engineering Designing what information enters an agent's context window at each step.
  15. Context window The maximum number of tokens a model can attend to in one call; the agent's working memory.
  16. Digital labour Agents treated as workforce capacity — measured in tasks completed, not licences sold.
  17. Evaluation (evals) Systematic tests that score an agent's outputs against expected results or a rubric.
  18. Frontier Firm Microsoft's term for an organisation with org-wide AI deployment and agents in active use.
  19. Guardrails Rules or classifiers that constrain what an agent may do, say or access.
  20. Human-in-the-loop (HITL) A checkpoint where a person must approve, edit or reject an agent's action before it proceeds.
  21. Jagged frontier The uneven capability profile of AI: superhuman on some tasks, failing on adjacent easy ones.
  22. LLM-as-judge Using a language model to grade another model's output against a rubric.
  23. MCP (Model Context Protocol) An open protocol that standardises how agents connect to tools, data and prompts.
  24. Multi-agent system Several agents with distinct roles coordinating on one task.
  25. Observability (agent tracing) Recording every step, tool call, token count and latency of an agent run so it can be inspected.
  26. Orchestrator The component (agent or code) that decomposes a task and delegates to sub-agents or tools.
  27. OSWorld The reference benchmark for computer-use agents: 369 real tasks on real operating systems.
  28. pass^k The probability an agent solves the same task k times in a row — a consistency metric.
  29. Prompt injection Untrusted content that manipulates an agent into following instructions its operator did not give.
  30. RAG (retrieval-augmented generation) Fetching relevant documents at query time and placing them in the model's context.
  31. ReAct A prompting pattern that interleaves reasoning traces with actions.
  32. Reasoning model A model trained to spend variable inference-time compute thinking before answering.
  33. Reliability (agent) How consistently an agent produces a correct outcome across runs and inputs, as opposed to peak capability.
  34. Sandboxing Running an agent's actions in an isolated environment so mistakes cannot reach production systems.
  35. Sub-agent An agent spawned by another agent to handle a bounded subtask in its own context.
  36. SWE-bench A benchmark of 2,294 real GitHub issues an agent must resolve so the repo's tests pass.
  37. System prompt Operator-level instructions that define an agent's role, rules and tools before the conversation begins.
  38. Time horizon (50%-task-completion) The length of task, in human-expert time, an agent completes with 50% success.
  39. Tool calling (function calling) The model emits a structured request to run a function; the runtime executes it and returns the result.
  40. Vibe coding Building software by describing intent to an AI and accepting its output without reading the code closely.
  41. τ-bench A benchmark of tool-using agents talking to simulated customers under policy constraints.

AI Coding Tools Glossary

49 terms · updated 2026-09-13 · companion to AI Coding Tools Statistics 2026.

  1. 'Almost right' problem AI output that is plausible and mostly correct but subtly wrong, so it costs more to find and fix the error than to write the code by hand.
  2. Acceptance rate The share of AI code suggestions a developer accepts; the most common but weakest measure of assistant value.
  3. Agent scaffold (harness) The non-model code around an LLM — tools, prompts, loop control, context management, retries — that turns a model into a working coding agent.
  4. Agentic coding A workflow in which the developer delegates whole tasks to a coding agent and reviews the result, rather than writing code with AI suggestions.
  5. AI code review Using an LLM to comment on, summarise or approve pull requests — the fastest-growing AI surface after code generation.
  6. AI coding assistant An IDE-integrated tool that suggests or completes code from an LLM while the developer stays in control of every edit.
  7. AI productivity paradox Individual output rises with AI coding tools while team- and company-level delivery metrics do not move.
  8. AI-accelerated technical debt Maintenance burden that accrues faster when code is generated quickly and reviewed lightly — duplication, inconsistent patterns, untested paths.
  9. AI-generated code Source code written by a model rather than a person; its share of commits is rising and its quality profile differs measurably from human code.
  10. Aider polyglot benchmark 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust, scored on correctness after two attempts, with cost per run reported.
  11. Augmentation vs automation Anthropic's classification of AI use: augmentation keeps the human in the loop (learning, iterating), automation has the AI complete the task.
  12. Benchmark contamination When benchmark tasks or their solutions appear in a model's training data, inflating scores without improving real capability.
  13. Benchmark saturation The point at which top models score near the ceiling of a benchmark so it can no longer distinguish them; HumanEval and SWE-bench Verified are saturated.
  14. Build-vs-buy shift Organisations choosing to build internal software with agentic coding tools instead of buying SaaS products.
  15. Code churn Lines that are rewritten or deleted shortly after being written (typically within two weeks); a proxy for code that was not right the first time.
  16. Code cloning (duplication) Copy-pasted or near-identical blocks of code; a maintainability cost that rose sharply with AI assistants.
  17. Code completion (inline suggestion) Real-time, ghost-text proposals of the next tokens, lines or blocks, shown as the developer types and accepted with a keystroke.
  18. Code LLM A large language model trained or fine-tuned predominantly on source code and developer text, optimised for generation, editing and reasoning about programs.
  19. Coding agent An LLM system that plans, edits files, runs commands and tests in a loop until a software task is done, with little or no human input per step.
  20. Context window The maximum number of tokens a model can attend to at once; it bounds how much of a codebase an agent can see without retrieval.
  21. Developer productivity metrics The measures used to evaluate AI coding tools: task completion time, throughput (PRs, tasks), cycle time, defect rate, and self-reported satisfaction.
  22. Directive interaction A session in which the user states the task and the AI completes it with minimal back-and-forth; the purest form of delegation.
  23. DORA metrics Four delivery measures — deployment frequency, lead time for changes, change failure rate, time to restore — used to judge whether AI adoption improves software delivery.
  24. Fill-in-the-middle (FIM) A training and prompting format that lets a code model complete text given both what comes before and after the cursor.
  25. HumanEval OpenAI's 2021 benchmark of 164 hand-written Python function problems with unit tests; the first standard code-generation eval, now saturated.
  26. METR developer RCT The 2025 randomised controlled trial in which 16 experienced open-source developers were 19% slower with AI tools on real issues while believing they were 20% faster.
  27. Model Context Protocol (MCP) An open protocol (Anthropic, Nov 2024) for exposing tools, data and prompts to LLM applications through a standard server interface.
  28. pass@k The probability that at least one of k sampled solutions is correct; pass@1 is a single attempt.
  29. Perception gap The difference between the productivity effect developers report and the one that is measured; consistently positive for AI tools.
  30. Public–private benchmark gap The drop in resolve rate between open-source benchmark tasks and tasks from unpublished commercial repositories.
  31. Pull-request agent A coding agent invoked from the issue tracker or PR that produces a reviewable pull request asynchronously, without a developer session.
  32. Reasoning model (extended thinking) A model that spends extra inference-time compute generating intermediate reasoning before answering, trading latency and cost for accuracy on hard tasks.
  33. Repository context (codebase awareness) Everything an assistant knows about the project beyond the open file: structure, dependencies, conventions, related code — gathered by indexing or on-demand search.
  34. Resolve rate The percentage of benchmark issues a system fixes such that all hidden tests pass; the headline metric for SWE-bench-style benchmarks.
  35. Retained code rate The share of AI-written characters or lines still present in the file after a period (e.g. 30 seconds, or at commit).
  36. Review bottleneck The point at which AI-accelerated code generation outpaces the team's capacity to review it, so cycle time stops improving.
  37. Security flaw rate The share of AI-generated code samples containing a security vulnerability; Veracode measured 45% across 100+ models with no improvement in newer ones.
  38. SWE-agent The open-source Princeton agent that introduced an agent-computer interface (ACI) for repositories; its 100-line successor scores 65% on SWE-bench Verified.
  39. SWE-bench A benchmark of 2,294 real GitHub issues from 12 Python repositories; a model passes if its patch makes the repository's tests pass.
  40. SWE-bench Pro Scale AI's harder successor to SWE-bench: 1,865 long-horizon tasks across 41 repositories, with public, held-out and private commercial subsets.
  41. SWE-bench Verified A 500-task subset of SWE-bench screened by human annotators to remove ambiguous or under-specified issues.
  42. Task horizon (time horizon) The length of a software task, measured in human expert time, that an AI can complete with a given success rate (usually 50%).
  43. Terminal (CLI) coding agent A coding agent that runs in the shell rather than an IDE — Claude Code, Codex CLI, Aider, Gemini CLI — operating on the whole repository and running commands directly.
  44. Terminal-bench A benchmark of tasks performed in a real shell — compiling, configuring, debugging, data processing — measuring agentic command-line competence.
  45. Test-time compute (parallel sampling) Running a model multiple times on the same task and selecting the best attempt, raising benchmark scores at extra cost.
  46. Token cost (cost per task) The dollar cost of a model run, priced per million input and output tokens; agentic coding multiplies it through long loops.
  47. Tool use (function calling) The model's ability to emit structured calls — run a command, read a file, search — that the harness executes and feeds back.
  48. Trust in AI-generated code Developers' reported confidence in the accuracy of AI output; usage keeps rising while trust falls.
  49. Vibe coding Building software by describing what you want to an AI and accepting its output without reading the code, relying on behaviour rather than inspection.