Buy me a coffee · $1
×

Support us with a small donation 💝

Every coffee keeps the site running. Thank you!

Models That See and Hear Changed What Gets Automated

 ·  1 min read

For most of their history, language models could only process text. That excluded an enormous amount of practical work, because the information businesses actually handle arrives as photographs, scans, screenshots, recordings and video.

What opened up

Document processing, first and largest. An invoice photographed on a phone, a handwritten form, a scanned contract – all previously required either manual entry or specialised systems built per document type.

Visual inspection. Comparing a photograph against a specification, spotting damage, checking that an installation matches a diagram.

Audio, which turned meetings, calls and interviews from things that had to be listened to into things that can be searched.

What still fails

Precise spatial reasoning. Models describe images well and reason about exact positions, distances and counts poorly. Asking how many of something appears in a photograph gives an unreliable answer.

Fine detail in dense images. Technical drawings, dense tables, and small text in photographs remain difficult.

And anything requiring measurement rather than description.

Where the value concentrates

In the unglamorous middle: taking information that exists in an awkward format and putting it into a structured one. That work was previously done by people, cost real money, and nobody enjoyed it.

The verification rule applies

Extracted data must be checkable. Cross-referencing a total against line items, validating a date format, comparing against an existing record – whatever the check is, it has to exist, because confident misreadings do happen.

What this means practically

If a process in your work involves someone retyping information from a picture or a document, that is now automatable. It is the most reliable value available from these tools and it gets discussed the least.