Skip to content
HUIAHUB

ifixai-ai/iFixAi

iFixAi

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

Metrics updated Established
View on GitHub (opens in a new tab)Homepage / demo

GitHub metrics

Stars
24K
growth after 7 days of data
Forks
1.4K
30-day growth
—
Open issues
33

The chart appears once the repository has been checked on two or more days.

What it does

iFixAi is an open-source Python tool that audits AI agents and models by running a set of test inspections and returning an A to F grade with a written report.

The free engine has 60 inspections. The grade comes from five core pillars: fabrication, manipulation, deception, unpredictability and opacity. Another 28 inspections from twenty premium categories are included and reported on their own, outside the grade. A grade counts as citable only when a judge model from a different vendor scores the run, so a full audit needs two API keys. You can test a bare model through providers such as Anthropic, OpenAI, Gemini, Azure, Bedrock and OpenRouter, or point the tool at your own agent's OpenAI-compatible HTTP endpoint.

There are three ways to run it: a setup wizard, a CLI with explicit flags for CI, or a plugin for Claude Code and Codex, with a scaffolded /ifixai-skill command for other agents. Reports land as JSON and Markdown in ./ifixai-results/. The README estimates judge costs for a full run with a Sonnet judge at roughly $12 to $18, billed by the provider. The company also sells iFixAi Pro, a paid audit service with up to 400 inspections. Telemetry is on by default outside CI and can be switched off.

Why it matters

The README argues that existing eval, red-teaming and observability tools focus on technical measures such as latency, token use and prompt injection. iFixAi instead checks whether an agent does the job it was given, for example whether it uses tools it was not granted, escalates privileges or drifts from instructions. It keeps the agent under test and the judge separate, pairing a judge from another vendor. The project claims an answer in less than 120 seconds.

Main features

  • 60 inspections grouped into 25 categories
  • A to F grade weighted across five core pillars
  • Judge from a different vendor for citable grades
  • Suites from `smoke` (3 tests) up to `all` (60)
  • Tests real agents over OpenAI-compatible HTTP endpoints
  • Plugin for Claude Code and Codex, plus a skill scaffold for other agents
  • JSON and Markdown reports in `./ifixai-results/`
  • Built-in mock provider to check the pipeline without API keys

Who it's for

  • Teams shipping AI agents that want an audit before release
  • Engineers who want agent checks running in CI
  • Risk and compliance staff who need a graded report to review
  • Claude Code or Codex users who prefer to start audits from inside the agent

Strengths

  • Separates the system under test from the judge so a model never grades itself.
  • Supports many providers and custom agents through a small adapter.
  • Marks inspections it cannot observe as insufficient_evidence instead of counting them as failures.
  • The Apache 2.0 licence covers both core and premium-tier inspections in the repository.

Limitations

  • Citable grades need two API keys, and judge calls are billed by the provider.
  • Full mode needs a hand-built fixture, and the default fixture ships with seeded defects that fail on purpose.
  • Telemetry is on by default outside CI.
  • Up to 400 inspections and the compliance reports sit in the paid iFixAi Pro service.

Getting started

Install the CLI with the extra for the provider you plan to test. A citable run also needs the SDK for a judge from another vendor, which the second command adds:

pip install "ifixai[openai]"
pip install "ifixai[anthropic,openai]"

Requirements: Python 3.10 or later, plus API keys for the model under test and, for citable grades, a judge from a different vendor.

Always follow the official instructions in the README on GitHub. This overview is our summary, not a copy.

Sources