Round-trip translation could help SciWard’s human voters review submissions written in invented languages. It can flag inconsistent translations, but even an exact match between the two English versions cannot establish that the first translation preserved what reviewers needed to see. That makes it a candidate for further research as a screening tool.
Why invented languages matter to SciWard
In my article on the Defense Panel, I explained how an invented language could conceal a prompt injection from the people who vote on SciWard’s funding disputes. A prompt injection is an instruction embedded in material the AI is supposed to assess, with the aim of influencing its behavior. The Defense Panel helps voters inspect suspicious material, while people make the final decision on account bans. SciWard Defense Panel · OWASP’s guide to prompt injection
My proposed rule is to exclude opaque invented-language material from AI-assisted funding evaluation. Yet a researcher may have a legitimate reason to introduce a formal language for expressing scientific claims. Could we make an exception when the author supplies two automatic translators, one into a language voters can read, such as English, and one back into the invented language?
What exact round-trip translation guarantees
Let I be the set of admissible texts in the invented language and E the set of English texts. Let F: I → E translate into English and G: E → I translate back. For a submission x, the simplest check is whether G(F(x)) = x.
If that equality holds for every admissible text, then F is injective. If two texts had the same English translation, applying G would recover the same original, so the two texts would have to be identical. This gives us a mechanical test for exact recovery without requiring a bijection with all English texts. The test still leaves one question open: does the English sentence express the right meaning? Even two translators that are exact inverses could agree on an incorrect mapping between meanings.
Why two matching English versions are insufficient
If back-translation changes the original wording, SciWard could try the chain I → E → I → E and compare the English texts e₁ = F(x) and e₂ = F(G(F(x))). If they match exactly, all we know is that F(G(e₁)) = e₁. The English text is unchanged by one more cycle.
Suppose two different invented-language texts, a and b, have different meanings under the language’s documented rules. Both translate to the same English sentence e, and translating e back always gives a. Starting from b yields b → e → a → e. The English versions are identical even though the distinction between a and b disappeared in the first step. Even a perfect English semantic comparison would accept this pair. Repeating the cycle would keep returning e.
What SciWard could investigate next
Research in machine translation has found that round-trip tests can help estimate translation quality. For SciWard, the question is whether this check helps voters spot omissions and changes in meaning when both translators come from the author. A study could use passages with independently established meanings and measure how often the check misses a material change or flags a faithful paraphrase. Research on round-trip translation evaluation · SciWard’s adversarial-testing plan
A possible exception for invented-language material would require an independent way to verify that the English translation preserves everything that could affect the funding review. The round-trip comparison can flag passages for closer inspection. A passing result alone gives SciWard insufficient grounds to admit the material.
Help Turn SciWard’s Beta into Evidence
SciWard remains experimental. The next proposed milestone is a five-month public adversarial test, with $1,000 planned for distribution to eligible funding recipients. Donations support the work required to run and evaluate that test.
Donate to the public test → Test proposal → Independent review → See how funds are planned to be used →”
AI-generated evaluations can contain factual errors or biases. Public testing, auditability, and external criticism remain part of the project.
