Advanced Error Reporting #1567
Replies: 8 comments 1 reply
-
|
@RobWelbourn Hi!
TypeBox actually accumulates any surplus properties in a import Schema from 'typebox/schema'
import Type from 'typebox'
Schema.Parse(Type.Object({
x: Type.Number(),
y: Type.Number(),
z: Type.Number(),
}, { additionalProperties: false }), {
x: 1,
y: 1,
z: 1,
w: 1 // <-- additional
})
// error: Uncaught (in promise) ParseError {
// schema: {
// type: "object",
// required: [ "x", "y", "z" ],
// properties: {
// x: { type: "number" },
// y: { type: "number" },
// z: { type: "number" }
// },
// additionalProperties: false
// },
// value: { x: 1, y: 1, z: 1, w: 1 },
// errors: [
// {
// keyword: "additionalProperties",
// schemaPath: "#",
// instancePath: "",
// params: { additionalProperties: [ "w" ] }, <-- additional
// message: "must not have additional properties"
// }
// ]
// }Ajv Compat | AlignmentThe TB 1.x error messages have been written to align closely to Ajv (which doesn't include the additional properties on the message either). Note that there is some consideration in 1.x to have TB be made compatible with current Ajv error reporting infrastructure. In this regard, TB tries to avoid generating anything above what a Ajv reporter might expect. import Type from 'typebox'
import * as Ajv from 'npm:ajv'
const ajv = new Ajv.Ajv()
ajv.validate(Type.Object({
x: Type.Number(),
y: Type.Number(),
z: Type.Number(),
}, { additionalProperties: false }), {
x: 1,
y: 1,
z: 1,
w: 1 // <-- additional
})
console.log(ajv.errors) // [
// {
// instancePath: "",
// schemaPath: "#/additionalProperties",
// keyword: "additionalProperties",
// params: { additionalProperty: "w" },
// message: "must NOT have additional properties"
// }
// ]There is still work to do on this aspect, but for 1.x, the goal is to be relatively inline with Ajv in terms of error reporting. Error MappingTypeBox does provide an option to override the error message via Locale settings, The following would loosely achieve this. import System from 'typebox/system'
import Schema from 'typebox/schema'
import Type from 'typebox'
System.Locale.Set((error) => {
// generate an error
const message = System.Locale.en_US(error)
// if additionalProperties, append with surplus (dynamic) property keys
return error.keyword === 'additionalProperties'
? `${message} (${error.params.additionalProperties.join(', ')})`
: message
})
Schema.Parse(Type.Object({
x: Type.Number(),
y: Type.Number(),
z: Type.Number(),
}, { additionalProperties: false }), {
x: 1,
y: 1,
z: 1,
w: 1
}) // { message: "must not have additional properties (w)" }I guess my preference would be to keep the errors as they are for now to ensure some compat with downstream Ajv packages, as well as to keep the door open for compiled errors in later versions of TypeBox (where the error message is generated from known schema properties only (not unknown / dynamic surplus properties)), but would be open to discussions all the same on how to improve the current 1.x infrastructure with consideration to the Ajv compat aspect. Would the "Error Mapping" approach shown above work for your use case? |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate the response, as always! I had missed the fact that the I've built a ParseError formatter that I modify from time to time, that currently looks like this: export function formatParseError(section: string | undefined, err: ParseError): string {
function cleanPath(path: string): string {
return path
.replaceAll('/', '.')
.replace(/^\./, '') // Remove leading dot if present
.replaceAll(/\.(\d+)/g, '[$1]'); // Convert .0, .1, etc. to [0], [1], etc.
}
// console.log('ParseError:', err);
const messages = [];
for (const e of err.cause.errors) {
const path = cleanPath((section ?? '') + e.instancePath);
let message = path
? path + ' ' + e.message
: e.message;
if (e.keyword === 'additionalProperties') {
message += ' ' + e.params.additionalProperties.join(', ');
}
messages.push(message);
}
return messages.join('\n');
}I'm building a configuration management library, so my goal here is to provide readily intelligible user feedback. Perhaps allowing user defined error messages might be an approach? |
Beta Was this translation helpful? Give feedback.
-
|
I must say I find handling errors with discriminated unions a bit challenging. Here's an excerpt from my JSON5 config file: {
// other stuff here
poll: {
servers: [
{
name: "Twilio (STUN)",
type: "foo", // Should be "stun" to be valid, but we want to test validation failure here
url: "global.stun.twilio.com:3478",
enabled: true
},
{
name: 'DB-IP.com',
type: 'json',
url: 'http://api.db-ip.com/v2/free/self',
attributeName: 'ipAddress',
enabled: true
},
],
},
}Here's the corresponding schema: const JsonServerSchema = Type.Object({
name: Type.String({ minLength: 1 }),
type: Type.Literal('json'),
url: Type.String({ format: 'url' }),
attributeName: Type.String({ minLength: 1, default: 'ip' }),
enabled: Type.Boolean({ default: true }),
}, {
additionalProperties: false,
});
const StunServerSchema = Type.Object({
name: Type.String({ minLength: 1 }),
type: Type.Literal('stun'),
url: Type.String({ pattern: '[a-zA-Z0-9.-]+(?::\d+)?' }),
enabled: Type.Boolean({ default: true }),
}, {
additionalProperties: false,
});
const PollerSchema = Type.Object({
interval: Type.Number({
description: 'Interval between IP address checks, in seconds',
minimum: 1,
default: 10,
}),
servers: Type.Array(
Type.Union([JsonServerSchema, StunServerSchema]),
{ minItems: 1 }
)
}, {
title: 'IP Address Poller',
description: 'See configuration file for available IP address servers.',
additionalProperties: false,
});...which results in the following validation errors: Now, this tells me the error is with the first server entry, but it is not exactly obvious what the problem is. What I would like is an error that says that the server type ...but we start having to do quite a bit of work in our formatter function. |
Beta Was this translation helpful? Give feedback.
-
|
@RobWelbourn Hi!, Custom Errors
Custom errors are still supported 1.x (albeit via a different mechanism using the https://github.com/sinclairzx81/typebox/blob/main/example/legacy/custom-errors.ts Union Errors
Yeah, unfortunately, Union validation errors continue to be a challenge. TB versions prior to 1.x had a few attempts at trying to offer something consistent, but downstream applications had a myriad of ways they wanted to present union errors, and I was never able to settle on a one size fits all error output. As of 1.x (and moving forward), TB will just generate a linear set of errors derived from the schema evaluation CHECK ORDER and will encode If it's helpful, the following is a quick draft formatted that groups anyOf sub-variant errors into indexed sets for the errors you provided. import { TLocalizedValidationError } from 'typebox/error'
// ------------------------------------------------------------------
// AnyOf Formatter:
//
// Groups AnyOf variants into sub group.
// ------------------------------------------------------------------
// seperate anyOf error into set[0], non-anyOf error into set[1].
function extractAnyOf(errors: TLocalizedValidationError[]): [TLocalizedValidationError[], TLocalizedValidationError[]] {
return errors.reduce<[TLocalizedValidationError[], TLocalizedValidationError[]]>((result, error) => {
return error.keyword === 'anyOf'
? [[...result[0], error], result[1]]
: [result[0], [...result[1], error]]
}, [[], []])
}
// for each anyOf error in set[0], group each subschema by variant index
function groupAnyOf(set: [TLocalizedValidationError[], TLocalizedValidationError[]]): TLocalizedValidationError[] {
return set[0].map(anyOf => {
const filtered = set[1].filter(schema => schema.schemaPath.startsWith(anyOf.schemaPath))
const variants = filtered.reduce<Record<string, TLocalizedValidationError[]>>((result, error) => {
const truncate = error.schemaPath.replace(anyOf.schemaPath + '/' + anyOf.keyword + '/', '')
const index = truncate.split('/')[0] ?? truncate // <-- variant index
return index in result
? { ...result, [index]: [...result[index], error] }
: { ...result, [index]: [error] }
}, {})
return { ...anyOf, variants }
})
}
// for each non-anyOf error in set[1], filter if not in set[0]
function ungroupAnyOf(set: [TLocalizedValidationError[], TLocalizedValidationError[]]): TLocalizedValidationError[] {
return set[1].filter((error) => !set[0].find(anyOf => error.schemaPath.startsWith(anyOf.schemaPath)))
}
// format anyOf errors
function formatAnyOf(errors: TLocalizedValidationError[]): TLocalizedValidationError[] {
const set = extractAnyOf(errors)
const grouped = groupAnyOf(set)
const ungrouped = ungroupAnyOf(set)
return [...grouped, ...ungrouped]
}
// ------------------------------------------------------------------
// Reference
// ------------------------------------------------------------------
const errors: TLocalizedValidationError[] = [
// errors as per reference
{
keyword: "required",
schemaPath: "#/properties/poll/properties/servers/items/anyOf/0",
instancePath: "/poll/servers/0",
params: { requiredProperties: ["attributeName"] },
message: "must have required properties attributeName"
},
{
keyword: "const",
schemaPath: "#/properties/poll/properties/servers/items/anyOf/0/properties/type",
instancePath: "/poll/servers/0/type",
params: { allowedValue: "json" },
message: "must be equal to constant"
},
{
keyword: "format",
schemaPath: "#/properties/poll/properties/servers/items/anyOf/0/properties/url",
instancePath: "/poll/servers/0/url",
params: { format: "url" },
message: 'must match format "url"'
},
{
keyword: "const",
schemaPath: "#/properties/poll/properties/servers/items/anyOf/1/properties/type",
instancePath: "/poll/servers/0/type",
params: { allowedValue: "stun" },
message: "must be equal to constant"
},
{
keyword: "anyOf",
schemaPath: "#/properties/poll/properties/servers/items",
instancePath: "/poll/servers/0",
params: {},
message: "must match a schema in anyOf"
},
// additional non-union errors
{
keyword: "const",
schemaPath: "#/properties/foo",
instancePath: "/poll/foo",
params: { allowedValue: "foo" },
message: "must be equal to constant"
},
{
keyword: "const",
schemaPath: "#/properties/bar",
instancePath: "/poll/bar",
params: { allowedValue: "bar" },
message: "must be equal to constant"
},
]
const formatted = formatAnyOf(errors)
console.dir(formatted, { depth: 10 })
// [
// {
// keyword: "anyOf",
// schemaPath: "#/properties/poll/properties/servers/items",
// instancePath: "/poll/servers/0",
// params: {},
// message: "must match a schema in anyOf",
// variants: {
// "0": [
// {
// keyword: "required",
// schemaPath: "#/properties/poll/properties/servers/items/anyOf/0",
// instancePath: "/poll/servers/0",
// params: { requiredProperties: [ "attributeName" ] },
// message: "must have required properties attributeName"
// },
// {
// keyword: "const",
// schemaPath: "#/properties/poll/properties/servers/items/anyOf/0/properties/type",
// instancePath: "/poll/servers/0/type",
// params: { allowedValue: "json" },
// message: "must be equal to constant"
// },
// {
// keyword: "format",
// schemaPath: "#/properties/poll/properties/servers/items/anyOf/0/properties/url",
// instancePath: "/poll/servers/0/url",
// params: { format: "url" },
// message: 'must match format "url"'
// }
// ],
// "1": [
// {
// keyword: "const",
// schemaPath: "#/properties/poll/properties/servers/items/anyOf/1/properties/type",
// instancePath: "/poll/servers/0/type",
// params: { allowedValue: "stun" },
// message: "must be equal to constant"
// }
// ]
// }
// },
// {
// keyword: "const",
// schemaPath: "#/properties/foo",
// instancePath: "/poll/foo",
// params: { allowedValue: "foo" },
// message: "must be equal to constant"
// },
// {
// keyword: "const",
// schemaPath: "#/properties/bar",
// instancePath: "/poll/bar",
// params: { allowedValue: "bar" },
// message: "must be equal to constant"
// }
// ]TypeBox Error FormattersI am actually quite open to providing a set of built in error formatters in 1.x under the For now, give the above a try and let me know how you go, would be happy to provide a couple more reference formatters if it's helpful. |
Beta Was this translation helpful? Give feedback.
-
|
I had also built a function to split errors into export function formatParseError(section: string | undefined, err: ParseError): string {
// Convert instance paths to dot notation and add square brackets for array indices.
function normalizePath(path: string): string {
return path
.replaceAll('/', '.')
.replace(/^\./, '') // Remove leading dot if present
.replaceAll(/\.(\d+)/g, '[$1]'); // Convert .0, .1, etc. to [0], [1], etc.
}
console.log('ParseError:', err.cause);
let errors = err.cause.errors;
const messages = [];
// Find all discriminated union errors.
const unionErrors = errors.filter(e => e.keyword === 'anyOf');
for (const unionError of unionErrors) {
// Group the errors by their instancePath.
const group = errors.filter(e => e.instancePath.startsWith(unionError.instancePath));
const path = normalizePath((section ?? '') + unionError.instancePath);
// Check for common types of error.
const requiredErrors = group.filter(e => e.keyword === 'required'); // Missing properties
const typeErrors = group.filter(e => e.keyword === 'type'); // Type mismatches
if (requiredErrors.length > 0) {
const message = requiredErrors[0].message;
messages.push(path ? `${path} ${message}` : message);
} else if (typeErrors.length > 0) {
const message = typeErrors[0].message;
messages.push(path ? `${path} ${message}` : message);
} else {
// Otherwise, treat as an invalid value of the discriminator.
// Get the allowed values for this group and create an error message.
const allowedValues = group
.filter(e => e.keyword === 'const')
.map(e => e.params.allowedValue ?? '');
const message = 'must be one of: ' + allowedValues.join(', ');
messages.push(path ? `${path} ${message}` : message);
}
// Remove the grouped errors from the main errors array to avoid duplicate messages.
errors = errors.filter(e => !group.includes(e));
}
// Handle other kinds of error.
for (const e of errors) {
const path = normalizePath((section ?? '') + e.instancePath);
let message = path ? `${path} ${e.message}` : e.message;
if (e.keyword === 'additionalProperties') {
message += ' ' + e.params.additionalProperties.join(', ');
}
messages.push(message);
}
return messages.join('\n');
}I'm sure it will evolve over time. |
Beta Was this translation helpful? Give feedback.
-
|
@RobWelbourn Nice :) ... but do see these formatting implementations can get a bit chaotic!! I actually spent a bit of time mulling over 1.x union error formatting last night, I do note that the previous // ------------------------------------------------------------------
// Reference
// ------------------------------------------------------------------
const T = Type.Union([
Type.Object({ x: Type.Number() }),
Type.Union([
Type.Object({ y: Type.Number() }),
Type.Object({ z: Type.Number() }),
])
])
const E = Value.Errors(T, {})
const R = formatAnyOf(E) // <--- previous implementation
console.dir(R, { depth: 100 })
// ------------------------------------------------------------------
// Current (Incorrect)
// ------------------------------------------------------------------
const A = [
{
keyword: "anyOf",
schemaPath: "#/anyOf/1",
instancePath: "",
params: {},
message: "must match a schema in anyOf",
variants: {
"0": [
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/0",
instancePath: "",
params: { requiredProperties: [ "y" ] },
message: "must have required properties y"
}
],
"1": [
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/1",
instancePath: "",
params: { requiredProperties: [ "z" ] },
message: "must have required properties z"
}
]
}
},
{
keyword: "anyOf",
schemaPath: "#",
instancePath: "",
params: {},
message: "must match a schema in anyOf",
variants: {
"0": [
{
keyword: "required",
schemaPath: "#/anyOf/0",
instancePath: "",
params: { requiredProperties: [ "x" ] },
message: "must have required properties x"
}
],
"1": [
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/0",
instancePath: "",
params: { requiredProperties: [ "y" ] },
message: "must have required properties y"
},
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/1",
instancePath: "",
params: { requiredProperties: [ "z" ] },
message: "must have required properties z"
}
]
}
}
]
// ------------------------------------------------------------------
// Better
// ------------------------------------------------------------------
const B = [
{
keyword: "anyOf",
schemaPath: "#",
instancePath: "",
params: {},
message: "must match a schema in anyOf",
variants: {
"0": [
{
keyword: "required",
schemaPath: "#/anyOf/0",
instancePath: "",
params: { requiredProperties: ["x"] },
message: "must have required properties x"
}
],
"1": [
{
keyword: "anyOf",
schemaPath: "#",
instancePath: "",
params: {},
message: "must match a schema in anyOf",
variants: {
"0": [
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/0",
instancePath: "",
params: { requiredProperties: [ "y" ] },
message: "must have required properties y"
},
],
"1": [
{
keyword: "required",
schemaPath: "#/anyOf/1/anyOf/1",
instancePath: "",
params: { requiredProperties: ["z"] },
message: "must have required properties z"
}
]
}
}
]
}
}
]I expect a formatter would need a recursive walk across the Let me know your thoughts, If you find the above reasonable, ill add it to a planned feature list and look at implementation sometime later in the year (possibly with another function |
Beta Was this translation helpful? Give feedback.
-
|
Yes, indeed , five minutes after I sent my previous message, I had to update the formatter to add a couple more types of error into the processing of Union types. 🙂 If I could describe the challenges I've had in building a formatter, they are these:
All this nitpicking aside, using a schema validator to process a config file is a massive win: it cuts down on writing boilerplate validation code. If the error messages from a validator are not as customized, that's perfectly OK. |
Beta Was this translation helpful? Give feedback.
-
|
@RobWelbourn Hi! Just a follow up on this. I think I will covert this issue to a discussion. I've been looking at better error reporting today, and there is a lot of ambiguity as to what TypeBox "should" report, and there's still not really a clear path to achieving a better output (specifications would really help here!!!) On top of the items you listed, some additional things I would like to see would be:
The current linear errors should provide a path to all of these, and I am currently looking at the "The ability to synthesize a TypeScript type definition from JSON Schema". For the hierarchical format, the main thing is that TypeBox encodes enough hierarchical / node information by way of the Updated AlgorithmThe following is the current algorithm I am looking at for the Linear > Hierarchical transformation. I am considering only generating a Node IF the schema happens to be a logical schema expression (allOf, anyOf, oneOf, etc), such that the hierarchical format is partially representative of a logical binary expression. // where Left and Right operands map to flat error sets.
const A = 1 | 2 // { type: 'Binary', 'op': '|', left: { const: 1 }, right: { const: 2 } }Here is the starter // ------------------------------------------------------------------
// PathTree
//
// Converts an array of paths into a Node tree. Each path is a
// schemaPath where the tree just encodes the hierarchical
// structure of the paths it finds.
//
// ------------------------------------------------------------------
interface Node {
path: string
nodes: Node[]
}
function PathTree(paths: string[]): Node {
const root: Node = { path: '', nodes: [] }
const sorted = [...paths].sort((a, b) => a.localeCompare(b))
const insert = (current: Node[], id: string): void => {
const parent = current.find(node => id.startsWith(node.path))
if (parent) {
insert(parent.nodes, id)
} else {
current.push({ path: id, nodes: [] })
}
};
sorted.forEach(id => insert(root.nodes, id))
return root
}
// ------------------------------------------------------------------
// HierarchicalErrors
//
// Restructures TLocalizedValidationError into a HierarchicalError
// Node based on the schemaPath. An implementation may only want
// to generate Node's for Logical Types (anyOf, allOf, oneOf), but
// here we generate for everything.
//
// ------------------------------------------------------------------
import { TLocalizedValidationError } from 'typebox/error'
interface HierarchicalError {
keyword: string
path: string
message: string
nodes: HierarchicalError[]
}
function CreateSchemaPathTree(errors: TLocalizedValidationError[]): Node {
const schemaPaths = errors.map(error => `${error.schemaPath}`)
const tree = PathTree(schemaPaths)
return tree.nodes[0]
}
function HierarchicalErrorNode(node: Node, errors: TLocalizedValidationError[]): HierarchicalError {
const error = errors.find(error => `${error.schemaPath}` === node.path)!
const path = error.schemaPath
const keyword = error.keyword
const message = error.message
// todo: add more information here
const nodes = node.nodes.map(node => HierarchicalErrorNode(node, errors))
return { keyword, path, message, nodes }
}
function HierarchicalErrors(errors: TLocalizedValidationError[]): HierarchicalError | null {
return (errors.length > 0)
? HierarchicalErrorNode(CreateSchemaPathTree(errors), errors)
: null
}
// ------------------------------------------------------------------
// Reference
// ------------------------------------------------------------------
import Type from 'typebox'
import Value from 'typebox/value'
const schema = Type.Union([
Type.Object({ x: Type.Number() }),
Type.Union([
Type.Object({ y: Type.Number() }),
Type.Object({ z: Type.Number() }),
])
])
const value = { }
const errors = Value.Errors(schema, value)
const result = HierarchicalErrors(errors)
console.log(errors)
console.dir(result, { depth: 100 })
// Linear: [
// {
// keyword: "required",
// schemaPath: "#/anyOf/0",
// instancePath: "",
// params: { requiredProperties: [ "x" ] },
// message: "must have required properties x"
// },
// {
// keyword: "required",
// schemaPath: "#/anyOf/1/anyOf/0",
// instancePath: "",
// params: { requiredProperties: [ "y" ] },
// message: "must have required properties y"
// },
// {
// keyword: "required",
// schemaPath: "#/anyOf/1/anyOf/1",
// instancePath: "",
// params: { requiredProperties: [ "z" ] },
// message: "must have required properties z"
// },
// {
// keyword: "anyOf",
// schemaPath: "#/anyOf/1",
// instancePath: "",
// params: {},
// message: "must match a schema in anyOf"
// },
// {
// keyword: "anyOf",
// schemaPath: "#",
// instancePath: "",
// params: {},
// message: "must match a schema in anyOf"
// }
// ]
// Hierarchical: {
// keyword: "anyOf",
// path: "#",
// message: "must match a schema in anyOf",
// nodes: [
// {
// keyword: "required",
// path: "#/anyOf/0",
// message: "must have required properties x",
// nodes: []
// },
// {
// keyword: "anyOf",
// path: "#/anyOf/1",
// message: "must match a schema in anyOf",
// nodes: [
// {
// keyword: "required",
// path: "#/anyOf/1/anyOf/0",
// message: "must have required properties y",
// nodes: []
// },
// {
// keyword: "required",
// path: "#/anyOf/1/anyOf/1",
// message: "must have required properties z",
// nodes: []
// }
// ]
// }
// ]
// }Will convert to an discussion, and rename the issue (as the original additionalProperties query should be resolved). Lets revisit this from time to time! Cheers |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for following up on this, Sir Clive! In terms of requirements, I'm building a library -- a side project of an app to do Dynamic DNS updates -- to load and validate configuration files, in JSON5 format, with command line support. As such, I need to provide feedback that app developers can relay to their users, about what's wrong with their configurations. I'm already using CSS strings to format the command line help, so some form of palette would likely be involved. Happy to provide input on requirements. I have a GitHub project on the go, which is not really in a releasable state, but here is a snapshot: https://github.com/RobWelbourn/Jorvik |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
ParseError includes the message
must not have additional propertieswhen the option{ additionalProperties: false }is specified, but unfortunately it doesn't say which property or properties were surplus.Beta Was this translation helpful? Give feedback.
All reactions