Skip to content

AAK-EU-AI-ACT-ART15-LOCALE-001: Multilingual user-facing agent lacks per-locale eval coverage

Severity
INFO
Category
Legal Compliance (legal-compliance)
Scanner
agent_audit_kit/scanners/eu_ai_act_art15_locale.py
Auto-fixable
No.
SARIF name
EuAiActArt15LocaleCoverage

What it detects

An agent config declares two or more locales (or multilingual: true) for a user-facing surface, but the repository's eval / test fixtures reference only a single language. Cross-lingual robustness is a required-evidence axis under EU AI Act Article 15 (binding for Annex III high-risk use cases on 2027-12-02 and for Annex I product-embedded high-risk systems on 2028-08-02), and Ford et al. 2026 (arXiv:2605.23157) document that single-language safety eval misses language-specific jailbreak / refusal-regression vectors that scale non-uniformly across languages.

How to fix

Add per-locale eval fixtures matching the declared locale set (one scenario file per language, or a parametrized eval matrix), or narrow the agent config's declared locales to the set you can evidence. The EU AI Act Article 15 evidence pack must show robustness testing across each user-facing language.

To confirm the finding is gone, report only this rule:

agent-audit-kit scan . --rules AAK-EU-AI-ACT-ART15-LOCALE-001

Limitations

Counts a locale as evaluated when its code appears in the path of any file under a directory named evals, fixtures, scenarios, i18n, locales, benchmarks and the like, whatever the file holds, so locales/fr.json, a file of UI strings, counts as an eval for French. It recognises 30 two-letter language codes (ta and te are not among them) and finds agent configs by file name (agent.yaml, crew.yaml, manifest.yaml and a few more; config.yaml is not read). A config file over 1 MB is not read.

References

No external references are recorded for this rule.


All rules. This page is generated from the rule registry when the documentation is built; to change it, change the rule in agent_audit_kit/rules/builtin.py. To give the rule a hand-written write-up, add docs/rules/AAK-EU-AI-ACT-ART15-LOCALE-001.md: the page then shows that text, with the registry reference appended under it.