Design lab: classifier evaluation
Rules-only local classifier against 30 hand-labeled development prompts. These were written by the same author who tuned the rules, so this shows consistency, not generalization; a blind holdout set is still needed. Scores are an uncalibrated ranking heuristic, never a probability.
- Fixtures passing
- 30 / 30
- Prefixes checked
- 90
- Premature transforms
- 0
- Stable threshold
- 3.5
A premature transform is a word-by-word prefix that would show confident controls for an intent the full question does not accept.
Confusion matrix (rows: expected, columns: predicted primary)
| Expected ↓ / Predicted → | Record | Document | Comparison | Data | Explanation | How-to | Navigation | Open question |
|---|---|---|---|---|---|---|---|---|
| Record | 7 | · | · | · | · | · | · | · |
| Document | · | 7 | · | · | · | · | · | · |
| Comparison | · | · | 3 | · | · | · | · | · |
| Data | · | · | · | 4 | · | · | · | · |
| Explanation | · | · | · | · | 4 | · | · | · |
| How-to | · | · | · | · | · | 2 | · | · |
| Navigation | · | · | · | · | · | · | 1 | · |
| Open question | · | · | · | · | · | · | · | 2 |
All prompts
| ID | Prompt | Expected | Got | Key slots | Result |
|---|---|---|---|---|---|
| sec-01 | AAPL | Record | Record · stable | {"recordId":"CIK0000320193","recordType":"company","entities":["Apple"]} | ✓ pass |
| sec-02 | $TSLA | Record | Record · stable | {"recordId":"CIK0001318605","recordType":"company","entities":["Tesla"]} | ✓ pass |
| sec-03 | Microsoft | Record | Record · ambiguous | {"recordId":"CIK0000789019","recordType":"company","entities":["Microsoft"]} | ✓ pass |
| sec-04 | CIK 320193 | Record | Record · stable | {"recordId":"CIK0000320193","recordType":"company","glossaryId":"SG-CIK"} | ✓ pass |
| sec-05 | 0000320193-25-000079 | Record | Record · stable | {"recordId":"0000320193-25-000079","recordType":"filing"} | ✓ pass |
| sec-06 | F | Record | Record · provisional | {"recordId":"CIK0000037996","recordType":"company","entities":["Ford Motor"]} | ✓ pass |
| sec-07 | Tesla 10-K | Document | Document · provisional | {"entities":["Tesla"],"documentType":"10-K"} | ✓ pass |
| sec-08 | Ford 10-K | Document | Document · provisional | {"entities":["Ford Motor"],"documentType":"10-K"} | ✓ pass |
| sec-09 | Walmart annual report | Document | Document · provisional | {"entities":["Walmart"],"documentType":"10-K"} | ✓ pass |
| sec-10 | Coca-Cola proxy statement | Document | Document · provisional | {"entities":["Coca-Cola"],"documentType":"DEF 14A"} | ✓ pass |
| sec-11 | Latest 8-K filings from JPMorgan | Document | Document · provisional | {"dateRange":{"relative":"latest"},"entities":["JPMorgan Chase"],"documentType":"8-K","recordType":"filing"} | ✓ pass |
| sec-12 | 10-K Find 10-Ks, or learn what one is | Document / Explanation | Document · provisional | {"documentType":"10-K"} | ✓ pass |
| sec-13 | What is a Form D? | Explanation | Explanation · provisional | {"documentType":"D"} | ✓ pass |
| sec-14 | What is a 10-K? | Explanation | Explanation · provisional | {"documentType":"10-K"} | ✓ pass |
| sec-15 | What is a CIK? | Explanation | Explanation · stable | {"glossaryId":"SG-CIK"} | ✓ pass |
| sec-16 | What does an accession number mean? | Explanation | Explanation · stable | {"glossaryId":"SG-ACCESSION"} | ✓ pass |
| sec-17 | Compare Apple and Microsoft revenue since 2020 | Comparison | Comparison · stable | {"dateRange":{"from":"2020"},"entities":["Apple","Microsoft"],"metrics":["revenue"],"comparands":[{"label":"Apple"},{"label":"Microsoft"}]} | ✓ pass |
| sec-18 | Compare AAPL and MSFT revenue since 2020 | Comparison | Comparison · stable | {"dateRange":{"from":"2020"},"entities":["Apple","Microsoft"],"metrics":["revenue"],"comparands":[{"label":"Apple"},{"label":"Microsoft"}]} | ✓ pass |
| sec-19 | Show Nvidia net income by year | Data | Data · stable | {"entities":["Nvidia"],"groupBy":"year","metrics":["net income"]} | ✓ pass |
| sec-20 | F revenue since 2021 | Data | Data · provisional | {"dateRange":{"from":"2021"},"entities":["Ford Motor"],"metrics":["revenue"]} | ✓ pass |
| sec-21 | revenue of tech companies in 2024 | Data | Data · provisional | {"dateRange":{"from":"2024","to":"2024"},"topic":"Technology","recordType":"company","metrics":["revenue"]} | ✓ pass |
| sec-22 | Compare Apple and Oracle revenue | Comparison | Comparison · stable | {"entities":["Apple"],"metrics":["revenue"],"comparands":[{"label":"Apple"},{"label":"Oracle"}]} | ✓ pass |
| sec-23 | How much did Pfizer earn in 2023? | Data | Data · provisional | {"dateRange":{"from":"2023","to":"2023"},"entities":["Pfizer"]} | ✓ pass |
| sec-24 | How do I submit a tip to the SEC? | How-to | How-to · stable | {"task":"submit-tip"} | ✓ pass |
| sec-25 | How do I get EDGAR access codes? | How-to | How-to · stable | {"glossaryId":"SG-EDGAR","task":"edgar-access"} | ✓ pass |
| sec-26 | Where is the full-text search? | Navigation | Navigation · stable | {"task":"search-filings","destinations":["full-text-search"]} | ✓ pass |
| sec-27 | T | Record | Record · provisional | {"recordId":"CIK0000732717","recordType":"company","entities":["AT&T"]} | ✓ pass |
| sec-28 | I want to see all the filings | Open question / Document | Document · provisional | {"documentType":"Filing","recordType":"filing"} | ✓ pass |
| sec-29 | Apple stock price | Open question | Open question · uncertain | {"entities":["Apple"]} | ✓ pass |
| sec-30 | Write me a poem | Open question | Open question · uncertain | {} | ✓ pass |