Score - TypeSafe AI

Use a Score When

Use a Score when the answer is a position on a spectrum you can describe in steps. For example, how severe a bug is, how happy a customer is, or how much Python experience a candidate has. If the answer is one of a fixed set of options with no order between them, use a Choice. If it’s a yes or no, use a Noul.

A Score answer is a position along your levels in score, which can fall between two levels. The model also returns a probability for every level in probabilities, and a confidence value for the answer.

Example Score Questions:

"How severe is the bug being reported?"
  → 0: Cosmetic; no impact to functionality
  → 1: Broken or degraded feature, but workaround exists
  → 2: Blocking issue; no workaround exists

"How formal is this outfit based on the description"
  → 0: gym clothes
  → 1: casual
  → 2: business casual
  → 3: formal
  → 4: black tie

"How relevant is this candidate's experience to the job posting"
  → 0: completely unrelated
  → 1: adjacent field
  → 2: some direct experience
  → 3: deep, direct experience

Request Structure

The POST request body to the TypeSafe API has the same three top-level fields as any other question type: state, which is the content to evaluate; model; and questions. Each Score question has the following fields:

• type: Always "score".
• instructions: The question the model answers. What it’s rating.
• criteria: An ordered array of level descriptions, from the low end of the scale to the high end. Needs at least two levels and takes up to 10.

Below is a request where the state is a bug report and the question is how severe the bug is:

{
  "state": "The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari.",
  "questions": {
    "bug_severity": {
      "type": "score",
      "instructions": "How severe is the reported issue?",
      "criteria": [
        "Cosmetic; no impact to functionality",
        "Broken or degraded feature, but workaround exists",
        "Blocking issue; no workaround exists"
      ]
    }
  }
}

Levels

Each entry in criteria is a level: one point on the spectrum of possible answers, described in words. A level’s number is its position in the criteria array, starting at 0, so the three entries above are levels 0, 1, and 2. The order of the array is the numbering.

The model gets the descriptions and nothing else, and each level is judged on its own against the state.

Reading a Score

Let’s look at how the score changes with different inputs. For example, using the question and its levels from the request above:

"How severe is the reported issue?"
  → 0: Cosmetic; no impact to functionality
  → 1: Broken or degraded feature, but workaround exists
  → 2: Blocking issue; no workaround exists

In these examples, confidence 1.0 means the returned distribution puts all its probability on one level. This describes the model’s answer, not a guarantee that the answer is correct.

Writing Good Levels

Describe situations, not degrees. “Broken or degraded feature, but workaround exists” gives the model something to match the state against. “Moderately severe” doesn’t. Concrete descriptions can help the model distinguish levels. Check the answers against known examples; higher confidence alone does not show that a description is better.