Answers from SEC filings that show their page.
A question-answering system over 360 real 10-K, 10-Q and 8-K filings. Every answer cites the filing and page it came from, or says the information isn’t there. Each part of it was kept only if a measured experiment on the FinanceBench benchmark showed it helped.
On 100 benchmark questions it never saw during tuning, it answers 49% correctly. Plain vector search manages 15%.
How much has the effective tax rate of Corning changed between FY2021 and FY2022?
Plain vector search
“The effective tax rate was 20.5% in FY2021 and 17.8% in FY2022.”
It read Corning’s 2019–2021 annual reports and a 2023 quarterly, none of them the FY2022 filing.
This system
“The effective tax rate was 20.2% in FY2021 and 22.9% in FY2022.”
It narrowed the search to Corning’s FY2022 10-K and cites pp. 77–78.
How it does on questions it never saw
FinanceBench’s 150 questions were split once: 50 for building and tuning, 100 held back. The held-back set was run a single time, after every decision was made.
| Plain vector search | This system | |
|---|---|---|
| Answered correctly | 15% | 49% |
| Right filing among the 5 passages read | 44% | 95% |
| Right page among them | 16% | 57% |
| Answers fully supported by their sources | 91% | 93% |
The improvement is +34 points, with a 95% confidence interval of +24 to +44 (paired bootstrap over the same questions). It also declined all 30 questions in a separate set built to be unanswerable, such as companies outside the corpus and years with no filing yet.
Every held-out question, answered
These are the real outputs from the held-out run, not cherry-picked examples. Open any question to compare both systems’ answers with the analyst’s, see which pages each one read, and see where the evidence actually was.
What was tried, and what was kept
Each experiment changed one thing and was compared with the previous best on the same 50 tuning questions. Changes that didn’t clearly help were dropped, including some that sound like best practice.
| Change | What happened | Decision |
|---|---|---|
| Filter by the company and fiscal year named in the question | Right page found: 20% → 38%. The largest single gain; most mistakes were the same company’s other years. | Kept |
| Rerank the top 50 passages with a cross-encoder | Together with the filter: right page found 54% | Kept |
| Add keyword search (BM25) to vector search | Right filing found fell from 66% to 48%: filings share too much boilerplate wording | Dropped |
| Smaller or page-bounded chunks | Worse or no change | Dropped |
| Parse tables as markdown | No gain, even when the model was handed the right page | Dropped |
| Let the model rewrite the search query | +6 points at the top 5, within noise | Not adopted |
| Make the model list figures before answering | Fewer refusals, same accuracy: the extra answers were wrong | Not adopted |
All experiments with confidence intervals and the evaluation methodology.
How an answer is made
Read the question
Rules pick out the company and fiscal year. They’re right on 50 of 50 tuning questions for the company and 48 of 50 for the year.
Search the right filings
Vector search over 81,517 passages, restricted to that company and year, returns the 50 closest passages. If nothing matches, the filter loosens.
Rerank
A cross-encoder (bge-reranker-v2-m3) reads each passage against the question and keeps the best 5.
Answer with citations
A local model (Qwen3 14B via Ollama) answers only from those passages and cites them. When they don’t contain the answer, it says so instead of guessing.
Check it
Every change is scored on 50 tuning questions. A gate on every push to GitHub re-checks retrieval quality, and fails the build if it drops.
Where it falls short
- The model is now the weak link. Even when handed the exact right pages, it answers 54% of the tuning questions correctly, and 18 of its 51 held-out misses happened with the right evidence in front of it.
- It’s cautious. It declines 26% of answerable questions, sometimes with the evidence already retrieved.
- It grades itself. The judge is the same local model as the answerer. The judge passed its sanity checks (98% or better on correctness), but a person hasn’t yet graded its grades.
- Reranking costs time: about 6–7 seconds per question on a laptop.