From Laya to s1-code: I trained a System One model to search code and built s1grep

About this work: this is the first model I have trained. I come from software engineering, not from AI research, so I worked with one simple rule: no decision without a measurement. Predictions were written before seeing the results, the final exams were opened only once, and this article also covers what did not work.

In short

• Laya, an open System One model, could not judge code: it got 1 of 20 searches right, the same as chance. We measured that in one afternoon, before building anything on top of it.
• We trained our own model, s1-code, in three versions. On its own, v3 gets 161 of 197 never-seen real searches right, against 144 for v1 (p = 0.002) and 152 for Qwen3-Embedding.
• We built s1grep, a local search tool in Rust: it ranks the right function first in 168 of 201 real searches, against 145 for embeddings alone, equally in English and Spanish.
• We took it from prototype to product over five releases (0.2 to 0.5): nine languages, large repositories searchable within minutes, keyword search where it helps, a background process nobody has to start, and a tool for coding agents. The exam held at 168 of 201 in every release.
• Not everything worked: v3 does not reliably beat v2, large GPU models are still better judges, and in other languages the judge helps Java but hurts JavaScript and Go. All of that is measured and reported here.

The question

Coding agents and people lose a lot of time looking for where something lives in a repository. grep finds words, not intentions: if you ask "where is a failed payment retried", the function may be called send_with_retry and contain none of the words in your question.

Tools like jevgrep solve this by sending your code to a model in the cloud. Our question was a different one:

Can we build a local, open and bilingual (English and Spanish) alternative that finds code by what it does, using System One models, and runs on an ordinary CPU?

The short answer is yes, with caveats. This article covers the 18 steps that led there, including what worked and what did not.

What a System One model is

A System One model does not write text. It takes a document and typed questions (pick an option, true or false, or a scale) and returns probabilities in a single pass. That makes it fast, it does not "hallucinate", and its answers can be calibrated: a 90% should be right 9 times out of 10.

s1-code takes a function and a question, and returns the probability that the function answers the search:

[CLS] noul question: This code answers the search: where is a failed payment retried [SEP]
      [MASK] no  [MASK] yes [SEP]
billing/gateway/client.py
GatewayClient.send_with_retry

def send_with_retry(self, document, attempts=3): ... [SEP]
→ yes: 0.93

We started from Laya (Convai Innovations, Apache 2.0), a 322-million-parameter multilingual System One model. s1-code is Laya fine-tuned for a single task: deciding whether a piece of code answers a search. Technically, it is a System One model used as a reranker.

Phase 1 · Measure before building

Does Laya understand code?

Before writing the search tool we ran a cheap test: 20 questions over 1,531 functions from private Python backend service repositories. A keyword search (BM25) picked 25 candidates and Laya ranked them using four different ways of asking.

Conclusion: do not build on Laya as it is. That 13-minute test saved weeks of work on a model that was not fit for this.

Do embeddings find the code?

We put together an exam of 53 real questions taken from commits and compared BM25, three embedding models, their fusion with BM25 and two rerankers, searching across every function in the repositories.

We also reviewed what already existed: there are open semantic code search tools (ck, osgrep and others), but none has a local System One judge trained for code. That was the gap we decided to fill.

Phase 2 · Getting the model out of Python

For the product to be a single binary, without Python, the model had to run on ONNX Runtime from Rust. Four findings:

• The official exporter was broken: it fixed the text length at 16 tokens and failed on anything longer. With PyTorch's dynamo exporter, ONNX matches the original on all 14 test questions.
• INT8 does not speed it up: dynamic quantization pushes every answer to 0.5, and weight-only quantization keeps the quality but is slower on a CPU. We kept float32.
• The Rust engine gives the same results as Python: identical tokens and probabilities within ±0.0006, on Windows too. Every new implementation gets its own parity test.
• The bfloat16 mistake: three models shipped in bfloat16, and the CPU has no instructions for that format. In float32 they index 5.8 times faster with the same result.

Phase 3 · Building the data

Training a judge takes thousands of "search → right function" pairs. We generated them "backwards": pick a function, and an open model (Qwen3-8B) writes the search someone who needs it would type, in English or Spanish and in one of six styles. A filter drops the ones that repeat the function name.

Reviewing two samples of 50 questions by hand, which cost cents, caught template phrases, leaked tests and generic questions. Fixing them before generating the 36,000 raised acceptance from 80% to 87%.

To teach it to say "no", every question comes with hard decoys: the most similar functions from the same repository that are not the right one.

Phase 4 · s1-code v1

Fine-tuning of multilingual Laya with its authors' recipe, 2 epochs on an NVIDIA L4 GPU in the cloud (93 minutes), and calibration on 1,000 held-out examples. Before training we set the bar, beating granite alone (35 of 53), and the goal: 44.

Then we built a bigger exam: 202 real questions split into dev (100, for making decisions) and test (102, opened only once). And we changed the rival: a cascade judge has to beat the strongest cheap search, Qwen3-Embedding, not the weakest. Against that rival, v1's advantage was no longer reliable.

Phase 5 · s1-code v2: more data, exact parity

v2 added sources with human-written questions and verified licenses: commit messages (CommitPackFT), issues linked to the fixed function (SWE-bench, SWE-Gym, SWE-smith), human searches (CoSQA) and functions from the organization's private repositories, filtered for secrets and personal data. Every question got a twin in the other language.

On validation, v2 looked far ahead (315 against 269 of 400). Breaking it down by question type revealed a shortcut: it scored 100% on CoSQA questions because it recognized the style of the right answer, not the search.

An honest verdict on v2: good research work and a tie as a model. What really improved was our ability to tell.

Phase 6 · s1-code v3: a teacher and memorization control

We analyzed v2's 25 failures: 12 had the right one in 2nd place and 9 confused it with a sibling method of the same class. We tried four teacher candidates on the same questions. The best was Qwen3-Reranker-4B (89 of 100 on dev); the most eye-catching, the 9B Nimble, came in below it. Picking the teacher with a test cost less than 1 USD.

v3 was trained with:

• 72,332 groups of 1 right answer and 7 decoys drawn from the real search (Qwen3-Embedding), with exact language parity.
• A loss that makes the right answer compete against its decoys.
• Teacher scores on 20,000 groups, mixed 50/50 with the real labels.
• A memorization check every 1,000 steps and early stopping.

To measure it, agents wrote a new test of 197 questions about functions that did not appear in any earlier exam. It was opened only once, with the prediction written beforehand.

Against the rule set beforehand: v3 had to reliably beat both v2 and Qwen3 alone. It beats Qwen3 alone (p = 0.0015) but not v2 (p = 0.65). The full rule was not met, just as the written prediction expected (about a 30% chance).

Phase 7 · s1grep: the product

With the model frozen, we built the search tool around it. It works in four steps, all on your CPU:

1. Index: tree-sitter splits each file into functions and methods; granite turns each one into a vector, stored in SQLite outside the repository. Only changed files are read again.
2. Search: the question becomes a vector and the 25 closest functions come out.
3. Judge: s1-code v3 reads the top 5 and answers with a probability.
4. Combine: the judge's ranking and the search ranking are fused with weights chosen on dev, never on test.

Speed

Cutting what the judge reads to 256 or 192 tokens makes it faster, but drops accuracy below not using it at all, because it was trained reading 384. What did work was judging the candidates one at a time instead of in a batch (25% faster, same answers) and keeping the models loaded in memory.

How to use it

s1grep setup                                     # download the models, once
s1grep "where is a failed payment retried" .     # search
s1grep index .                                   # index a large repository and follow its progress
s1grep status                                    # what is loaded and how much is indexed
s1grep mcp                                       # tools for agents: search_code and index_status
s1grep skill --install                           # usage guide for Claude Code

It is a single binary for Linux and Windows, without Python. Nobody has to start or stop anything: the first search brings up a background process that only listens on your machine, with a key only your user can read, and shuts down after 30 minutes without use.

Phase 8 · Does it work for other languages?

s1-code was trained on Python only. To find out whether it works in other languages, we built an exam from CodeSearchNet: 100 searches per language over public repositories, with granite bringing the candidates from the same repository.

Our prediction, written beforehand, said no language would end up below granite alone. It failed in two. The first version of s1grep was limited to Python; since 0.2.6 it reads nine languages with the same judge, and this table is published so everyone knows what to expect. The real fix is a judge trained on several languages.

Phase 9 · From prototype to product (s1grep 0.2 to 0.5)

Version 0.1 proved the idea, but on a large repository the first search could take minutes, a server had to be started by hand, and it only understood Python. Before adding more features, we wrote down the product architecture, its documented decisions, and one rule for every change: the 201-search exam cannot go down. Over the next five releases it held at 168.

Large repositories: searchable within minutes

Indexing the full code of each function runs at about 5 to 10 per second on a CPU: a 25,000-function repository takes hours. The fix was to index in two passes. First an outline of each function (path, name and first lines), which is enough to search the whole repository; then, in the background, the full code, which is what gives the final quality. Searches answer from the first moment and report how much of the repository is searchable so far.

For outlines, a model three times smaller performs almost the same, so s1grep uses both: the small one for the first pass and the main one for the final pass. With the full repository indexed, search is exactly what it was before.

Nine languages

Each language is a tree-sitter query of a few lines that says what a function is and what its container is called (a class, a Rust impl, the receiver of a Go method). Adding a language does not touch the rest of the code. Python keeps the extractor the model was trained and examined with, so its results do not change.

Keyword search only where it helps

Search by meaning fails on what people look up literally: an exact name (parse_config) or an error message copied from a log. For that, each repository also has a keyword index (BM25), with a tokenizer built for code that splits camelCase and snake_case and does not depend on any human language.

Using it on every search made the exam worse: a Spanish question shares few words with code written in English, so the keyword list only added noise. The published rule decides by the shape of the query: if it looks like code, all of its words count; if it is a sentence, only exact matches do.

A judge that stops when it is sure

The judge reads the candidates in order and each read costs about 0.4 seconds. If one goes above 90% probability, there is no need to read the rest.

For coding agents

The background process uses two threads: one answers instantly (whether it is alive, its status, stopping it) even while the other is searching or indexing, and a search abandoned by its client is cancelled. Over MCP, agents get two tools: search_code, with paging, coverage notes and progress notifications, and index_status. The protocol, the architecture decisions and the exams are documented in the repository.

What we tried and dropped

Every idea was measured on the same exam before going in. These did not make it:

The second to last deserves a note: it is easy to design a rule while looking at the search that failed and convince yourself it improves the product. It fixed that case and broke 27 correct answers elsewhere. That is why no rule goes in without passing the exam.

Lessons on method

1. Measure before building. A 20-question test avoided building on a model that was not fit for the job.
2. Every new implementation needs its own parity test. ONNX, Rust, Windows and INT8 all "loaded without errors"; only comparing against the original showed which ones gave the same results.
3. The exam must not be written while reading the answer. Questions written while looking at the code favored keyword search.
4. Review samples before scaling. Two samples of 50 questions fixed defects that would have contaminated 36,000.
5. A judge has to beat the strongest cheap rival. Beating the weakest looked like a success and was not.
6. Write the prediction before the result and open the test only once. With 100 questions, a 9-answer difference can be chance.
7. Break it down before celebrating. A jump from 269 to 315 came entirely from a dataset shortcut.
8. Watch memorization during training, not afterwards, and keep the best checkpoint.
9. Pick the teacher with a test. The most eye-catching model was not the best.
10. Design for the cloud failing. The provider shut down GPUs twice in the middle of a run; saving every result as it arrived turned that into minutes lost, not hours.
11. A rule that fixes one case can break forty. Every improvement is measured on the full exam, never on the example that inspired it.
12. Measure what you shipped, not what you think you shipped. A value changed by mistake left outlines at 2 lines for two releases; only the exam caught it.
13. A model's published numbers do not replace your own exam. A model with better results on public code benchmarks lost 14 searches on ours.

Limits and costs

• The exams have 200 questions: they detect differences of about 10 answers, not smaller ones.
• If the search does not bring the right function among the candidates, the judge cannot rescue it.
• Large GPU models are still better judges; s1-code wins on size, CPU, language parity, license and privacy.
• On a large repository everything becomes searchable within minutes, but the final quality (the full code of each function) takes longer: about 5 to 10 functions per second on a CPU.
• The judge was trained on Python: in other languages and in highly domain-specific code it gives low probabilities, and it cannot yet reliably say that the answer is not in the repository.
• The private repositories used for the exam and part of the training are not published; they are described only in aggregate.
• We have not yet measured the effect on a real agent (tokens or time saved), which is the comparison jevgrep makes.

What comes next

• s1-code v4: a generalist judge trained on several languages from the start, with the same method and the same exam rules.
• Being able to say "it is not here": an exam with questions whose answer does not exist in the repository, and a judge calibrated to recognize them. That is what would matter most to an agent: to stop searching where there is nothing.
• Measuring with real agents, on the same tasks jevgrep uses: how many files an agent reads and how many tokens it spends with and without s1grep.

Each version is published only if it beats the previous one on the exam. If it does not, it gets reported anyway.

Resources

References

• Laya (Convai Innovations, Apache 2.0), the base model.
• jevgrep and layagrep, the reference tools.
• granite-embedding-278m-multilingual and granite-embedding-97m-multilingual-r2 (IBM), and Qwen3 Embedding and Reranker.
• tree-sitter and tantivy, for reading code and keyword search.
• CommitPackFT, SWE-bench, SWE-Gym, SWE-smith, CoSQA and CodeSearchNet.