Gunnar Wolf: As Answers Get Cheaper, Questions Grow Dearer

This post is an unpublished review

for As Answers Get Cheaper, Questions Grow Dearer

This opinion article tackles the much discussed issues of Large Language
Models (LLMs) both endangering jobs and improving productivity.

The authors begin by making a comparison, likening the current
understanding of the effects LLMs are currently having upon
knowledge-intensive work to that of artists in the early XIX century, when
photography was first invented: they explain that photography didn’t result
in painting becoming obsolete, but undeniably changed in a fundamental
way. Realism was no longer the goal of painters, as they could no longer
compete in equal terms with photography. Painters then began experimenting
with the subjective experiences of color and light: Impressionism no longer
limits to copying reality, but adds elements of human feeling to creations.

The authors argue that LLMs make getting answers terribly cheap — not
necessarily correct, but immediate and plausible. In order for the use of
LLMs to be advantageous to users, a good working knowledge of the domain in
which LLMs are queried is key. They cite as LLMs increasing productivity on
average 14% at call centers, where questions have unambiguous answers and
the knowledge domain is limited, but causing prejudice close to 10% to
inexperience entrepreneurs following their advice in an environment where
understanding of the situation and critical judgment are key. The problem,
thus, becomes that LLMs are optimized to generate plausible answers. If
the user is not a domain expert, “plausibility becomes a stand-in for
truth”. They identify that, with this in mind, good questions become
strategic: Questions that continue a line of inquiry, that expand the
user’s field of awareness, that reveal where we must keep looking. They
liken this to Clayton Christensen’s 2010 text on consulting¹: A
consultant’s value is not in having all the answers, but in teaching
clients how to think.

LLMs are already, and will likely become more so as they improve,
game-changing for society. The authors argue that for much of the 20th
century, an individual’s success was measured by domain mastery, but bring
to the table that the defining factor is no longer knowledge accumulation,
but the ability to formulate the right questions. Of course, the authors
acknowledge (it’s even the literal title of one of the article’s sections)
that good questions need strong theoretical foundations. Knowing a specific
domain enables users to imagine what should happen if following a specific
lead, anticipate second-order effects, and evaluate whether plausible
answers are meaningful or misleading.

Shortly after I read the article I am reviewing, I came across a data point
that quite validates its claims: A short, informally published paper on
combinatorics and graph theory titled “Claude’s Cycles”² written by Donald
Knuth (one of the most respected Computer Science professors and
researchers and author of the very well known “The Art of Computer
Programming” series of books). Knuth’s text, and particularly its
“postscripts”, perfectly illustrate what the article of this review
conveys: LLMs can help a skillful researcher “connect the dots” in very
varied fields of knowledge, perform tiring and burdensome calculators, even
try mixing together some ideas that will fail — or succeed. But guided by a
true expert of the field, asking the right, insightful and informed
questions will the answers prove to be of value — and, in this case, of
immense value. Knuth writes of a particular piece of the solution, “I would
have found this solution myself if I’d taken time to look carefully at all
760 of the generalizable solutions for m=3”, but having an LLM perform all
the legwork it was surely a better use of his time.

¹ Christensen, C.M. How Will You Measure Your
Life?
Harvard Business Review Press (2017).

² Knuth, D. Claude’s Cycles. https://cs.stanford.edu/~knuth/papers/claude-cycles.pdf