This is taken from my danish blog. It's a kind of a paradox. I thought of it in relation to language models and poetry writing. It goes as thus:
If we have no formal understanding of what a good poem is, we have to be told which poems are good and which are bad. Maybe told by a reviewer of poetry. In the most extreme case what is good vs. what is bad is totally up to some reviewer. He or she says something is good, and we blindly reproduce this position. In this situation a machine might as well write good poetry. Put differently: a good poem can be a random sequence of words.
In order to understand what is good about a poem we need some way of formulating it - some formal framework stating what works and what doesn't when writing good poetry. With an advanced enough formal framework we can automate the process of writing good poetry.