cmplib 1.4.1 · Michał Góral

I released cmplib 1.4.1. With 1.4.x my focus was on improving how cmplib reports errors, especially in pytest. All matchers now have shorter and more meaningful representations with assertions reporting only failed matchers. (sidenote: In practice, because of short-circuiting, only the first one is reported.)

In short, assertion messages change from this:

assert messages == Seq(
            Object(role="user", parts=[Object(text="msg2")]),
            Object(role="assistant", parts=Not(IsEmpty())),
            Object(role="user", parts=[Object(text="msg4")]),
            Object(role="assistant", parts=Not(IsEmpty())),
            Object(role="user", parts=[Object(text="msg5e")]),
            Object(role="assistant", parts=Not(IsEmpty())),
        )
E       assert [Message(role='user', parts=[TextPart(text='msg2', provider_metadata=None)], provider_metadata=None), Message(role='assistant', parts=[TextPart(text='{"prompt": "msg2", "system": "", "attachments": [], "stream": true, "messages": [{"role": "user", "parts": [{"type": "text", "text": "msg2"}]}], "options": {"example_bool": null, "indent": null, "thinking": null}, "tool_results": []}\n', provider_metadata=None)], provider_metadata=None), Message(role='user', parts=[TextPart(text='msg4', provider_metadata=None)], provider_metadata=None), Message(role='assistant', parts=[TextPart(text='{"prompt": "msg4", "system": "", "attachments": [], "stream": true, "messages": [{"role": "user", "parts": [{"type": "text", "text": "msg2"}]}, {"role": "assistant", "parts": [{"type": "text", "text": "{\\"prompt\\": \\"msg2\\", \\"system\\": \\"\\", \\"attachments\\": [], \\"stream\\": true, \\"messages\\": [{\\"role\\": \\"user\\", \\"parts\\": [{\\"type\\": \\"text\\", \\"text\\": \\"msg2\\"}]}], \\"options\\": {\\"example_bool\\": null, \\"indent\\": null, \\"thinking\\": null}, \\"tool_results\\": []}\\n"}]}, {"role": "user", "parts": [{"type": "text", "text": "msg4"}]}], "options": {"example_bool": null, "indent": null, "thinking": null}, "tool_results": []}\n', provider_metadata=None)], provider_metadata=None), Message(role='user', parts=[TextPart(text='msg5', provider_metadata=None)], provider_metadata=None), Message(role='assistant', parts=[TextPart(text='{"prompt": "msg5", "system": "", "attachments": [], "stream": true, "messages": [{"role": "user", "parts": [{"type": "text", "text": "msg2"}]}, {"role": "assistant", "parts": [{"type": "text", "text": "{\\"prompt\\": \\"msg2\\", \\"system\\": \\"\\", \\"attachments\\": [], \\"stream\\": true, \\"messages\\": [{\\"role\\": \\"user\\", \\"parts\\": [{\\"type\\": \\"text\\", \\"text\\": \\"msg2\\"}]}], \\"options\\": {\\"example_bool\\": null, \\"indent\\": null, \\"thinking\\": null}, \\"tool_results\\": []}\\n"}]}, {"role": "user", "parts": [{"type": "text", "text": "msg4"}]}, {"role": "assistant", "parts": [{"type": "text", "text": "{\\"prompt\\": \\"msg4\\", \\"system\\": \\"\\", \\"attachments\\": [], \\"stream\\": true, \\"messages\\": [{\\"role\\": \\"user\\", \\"parts\\": [{\\"type\\": \\"text\\", \\"text\\": \\"msg2\\"}]}, {\\"role\\": \\"assistant\\", \\"parts\\": [{\\"type\\": \\"text\\", \\"text\\": \\"{\\\\\\"prompt\\\\\\": \\\\\\"msg2\\\\\\", \\\\\\"system\\\\\\": \\\\\\"\\\\\\", \\\\\\"attachments\\\\\\": [], \\\\\\"stream\\\\\\": true, \\\\\\"messages\\\\\\": [{\\\\\\"role\\\\\\": \\\\\\"user\\\\\\", \\\\\\"parts\\\\\\": [{\\\\\\"type\\\\\\": \\\\\\"text\\\\\\", \\\\\\"text\\\\\\": \\\\\\"msg2\\\\\\"}]}], \\\\\\"options\\\\\\": {\\\\\\"example_bool\\\\\\": null, \\\\\\"indent\\\\\\": null, \\\\\\"thinking\\\\\\": null}, \\\\\\"tool_results\\\\\\": []}\\\\n\\"}]}, {\\"role\\": \\"user\\", \\"parts\\": [{\\"type\\": \\"text\\", \\"text\\": \\"msg4\\"}]}], \\"options\\": {\\"example_bool\\": null, \\"indent\\": null, \\"thinking\\": null}, \\"tool_results\\": []}\\n"}]}, {"role": "user", "parts": [{"type": "text", "text": "msg5"}]}], "options": {"example_bool": null, "indent": null, "thinking": null}, "tool_results": []}', provider_metadata=None)], provider_metadata=None)] == (is len of 6 AND obj[0]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg2']) AND obj[1]: (obj.role: 'assistant' AND obj.parts: !is empty) AND obj[2]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg4']) AND obj[3]: (obj.role: 'assistant' AND obj.parts: !is empty) AND obj[4]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg5e']) AND obj[5]: (obj.role: 'assistant' AND obj.parts: !is empty))
E        +  where (is len of 6 AND obj[0]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg2']) AND obj[1]: (obj.role: 'assistant' AND obj.parts: !is empty) AND obj[2]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg4']) AND obj[3]: (obj.role: 'assistant' AND obj.parts: !is empty) AND obj[4]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg5e']) AND obj[5]: (obj.role: 'assistant' AND obj.parts: !is empty)) = Seq((obj.role: 'user' AND obj.parts: [obj.text: 'msg2']), (obj.role: 'assistant' AND obj.parts: !is empty), (obj.role: 'user' AND obj.parts: [obj.text: 'msg4']), (obj.role: 'assistant' AND obj.parts: !is empty), (obj.role: 'user' AND obj.parts: [obj.text: 'msg5e']), (obj.role: 'assistant' AND obj.parts: !is empty))
E        +    where (obj.role: 'user' AND obj.parts: [obj.text: 'msg2']) = Object(role='user', parts=[obj.text: 'msg2'])
E        +    and   (obj.role: 'assistant' AND obj.parts: !is empty) = Object(role='assistant', parts=!is empty)
E        +      where !is empty = Not(is empty)
E        +        where is empty = IsEmpty()
E        +    and   (obj.role: 'user' AND obj.parts: [obj.text: 'msg4']) = Object(role='user', parts=[obj.text: 'msg4'])
E        +    and   (obj.role: 'assistant' AND obj.parts: !is empty) = Object(role='assistant', parts=!is empty)
E        +      where !is empty = Not(is empty)
E        +        where is empty = IsEmpty()
E        +    and   (obj.role: 'user' AND obj.parts: [obj.text: 'msg5e']) = Object(role='user', parts=[obj.text: 'msg5e'])
E        +    and   (obj.role: 'assistant' AND obj.parts: !is empty) = Object(role='assistant', parts=!is empty)
E        +      where !is empty = Not(is empty)
E        +        where is empty = IsEmpty()

to this:

>       assert messages == Seq(
            Object(role="user", parts=[Object(text="msg2")]),
            Object(role="assistant", parts=Not(IsEmpty())),
            Object(role="user", parts=[Object(text="msg4")]),
            Object(role="assistant", parts=Not(IsEmpty())),
            Object(role="user", parts=[Object(text="msg5e")]),
            Object(role="assistant", parts=Not(IsEmpty())),
        )
E       AssertionError: assert list matches:
E         obj[4]: (obj.role: 'user' AND obj.parts: [obj.text: 'msg5e'])
E         (only first failed matcher printed)

This is automatically enabled for pytest (cmplib now ships with a pytest plugin), but you can disable it with --cmplib-no-explain pytest flag. For non-pytest code, you may use new assert_eq() and assert_ne() functions, or use a low-level explain_failures(lhs, op, rhs) function, which accepts the partitioned expression (with operator as a string and exact instances of used operands):

validator = And(1, Or(Gt(2), Lt(1)))
if not foo == validator:  # or `foo != validator`, but this is counter-intuitive
    print("validation failure:", explain_failures(foo, "==", validator))

New matchers

cmplib ships 2 new matchers. First one is Seq(), which should be used to match sequences. It aims to replace code like this to enable the nice reporting:

assert messages == [ Object(...), Object(...), ... ]

The other matcher is Falsy(), which is a counterpart for existing Truish() matcher.

Future breaking changes

Naming the Seq matcher was a headache: all good names were already taken and I wanted to avoid a collision with typing.Sequence. Items and Values matchers exist, but their usefulness seems limited, especially the Values() matcher, which, considering its name, checks I-don’t-know-what. In future I might remove/rename the current Values matcher and reuse its name for different purpose. Clear, coherent and self-describing interface will be my target for next big release (2.0).