Build, Buy or Partner

Whether to run data operations in-house or outside is usually argued on cost. A fifty-year-old theory of the firm gives a better answer, and it depends on the kind of data.

In June 2025, Meta agreed to invest about 14.3 billion dollars for a 49% stake in Scale AI, a company whose business was supplying labelled data and human feedback to AI developers. Scale’s chief executive joined Meta. Within days, according to Reuters and other outlets, several of Scale’s largest customers, themselves competitors of Meta, began moving work elsewhere.

Their concern was easy to understand. A data supplier sees what a model developer is working on. It learns which capabilities the developer is trying to improve, which failures it is trying to fix, how it defines quality and what its evaluation looks like. That information is strategically sensitive, and it had just come under the partial ownership of a rival.

The episode was unusual in scale. The underlying question is one every organisation building with AI has to answer. Which data work should be done inside the company, which bought from a supplier, and on what terms?

A theory that fits

The standard economic account of why firms make some things and buy others was developed by Oliver Williamson, building on Ronald Coase, and earned Williamson a Nobel prize. Its central idea is that using a market has costs beyond the price: the cost of specifying what you want, of checking that you received it, and of protecting yourself if the other party behaves opportunistically.

Williamson identified the conditions under which those costs become high enough that it is better to bring the activity inside. Two matter most here.

Asset specificity. How far does the work depend on knowledge, tools or relationships particular to this buyer? A highly specific investment has little value in any other use, which leaves each side dependent on the other.

Measurement difficulty. How easily can the buyer verify the quality of what was delivered? Where quality is hard to observe, a supplier has room to cut corners, and a contract cannot fully prevent it.

When both are low, the market works well. When both are high, firms tend to integrate, or to form long-term relationships that sit between a market and a hierarchy.

AI data spans the whole range, and that explains why one answer does not fit.

Four kinds of data work

Generic and easy to verify. Bounding boxes around cars. Transcription of clear audio in a major language. The task is the same for every buyer, guidelines are standard and correctness can be checked by sampling. This is a commodity. Buy it, competitively, and expect the price to keep falling as automation absorbs it.

Generic and hard to verify. Preference ranking of general-purpose assistant responses. Open-ended quality ratings. Any buyer could use the output, but whether it is good is hard to tell from the output alone. Here the supplier’s process, its recruitment, training and quality control, is what is being bought, and reputation and audit rights matter more than price.

Specific and easy to verify. Extracting fields from a company’s own invoice formats. The task is particular to the buyer, but a correct answer is unambiguous. This can be bought, with an investment in transferring the specification. Lock-in is the risk: the supplier accumulates knowledge of the formats that the buyer would have to rebuild elsewhere.

Specific and hard to verify. Judging whether an underwriting assistant’s recommendation is sound by the insurer’s own standards. Evaluating a clinical summarisation model against one hospital’s practice. Red-teaming a product for the ways its particular users might be harmed. The knowledge needed is inside the organisation, and so is the ability to tell good work from bad.

The fourth quadrant is where arm’s-length purchasing breaks down, and it is where most of the value in applied AI now lies.

What the research says about the boundary

Studies of annotation as a working practice support the theory.

Miceli, Schuessler and Yang conducted fieldwork inside annotation companies and found that the meaning of the labels was largely set by the client. Annotators applied categories they had been given, often without the context to question them, and client instructions were treated as ground truth even when annotators could see they fitted the data poorly. The authors’ title describes what they observed: interpretation was imposed downward. Knowledge that annotators had about the data did not flow back up to those who designed the scheme.

Wang, Prabhakaran and Díaz interviewed annotation workers in India and reached a related conclusion. Annotators were treated as interchangeable executors of a specification, while often holding knowledge that would have improved it.

Sambasivan’s data-cascades study listed poor documentation across organisational boundaries among its four main triggers. Assumptions made on one side of a handover were invisible on the other.

These findings point in the same direction. The organisational boundary between whoever understands the purpose of the data and whoever produces it is where information is lost. A pure outsourcing arrangement maximises that boundary.

The case for building

The argument for doing data work internally follows directly.

Knowledge stays close to the work. The people defining quality are near the people labelling. Guidelines change as understanding improves, without contract variations.

Confidentiality. Sensitive data, and the sensitive knowledge of what the organisation is trying to improve, never leaves. The 2023 reports of Samsung employees pasting proprietary code into a public chatbot were a reminder of how easily information escapes through tools, and a supplier’s workforce is a larger surface.

Accumulated expertise. Annotators who stay on a domain become skilled in it. That is an asset, and internally it belongs to the organisation.

Feedback. Problems found in labelling inform the product. Production failures inform the labelling.

The case against building

The costs are equally real.

It is a different business. Recruiting, training, scheduling and managing an annotation workforce, building tooling and running quality systems is an operational discipline. Most AI teams are not set up for it, and those that try tend to underinvest and produce the low-status, under-resourced data function that Sambasivan’s interviewees described.

Demand is uneven. Data needs arrive in bursts, around a new model, a new product, a new language. A permanent team sized for peaks is idle between them. One sized for the average is overwhelmed.

Breadth is hard to hold. A project may need radiologists this quarter, Tamil speakers next quarter and security researchers after that. No single employer can retain all of them.

Tooling and method move quickly. Specialists who work across many clients see more problems and more ways of solving them.

A team that sends its domain experts to label data full-time has also made an expensive choice about how to use its scarcest people.

The intermediate form

Williamson’s theory predicts that when specificity and measurement difficulty are high but full integration is impractical, firms form long-term, relationship-based arrangements with safeguards. That is a fair description of what works for data.

In such a partnership the client keeps certain things. It owns the definition of quality: the taxonomy, the rubric, the evaluation set. It supplies domain experts for the decisions only they can make, such as guideline design, adjudication of hard cases and the gold standard. It retains the data and the rights to it.

The partner brings what is difficult to build internally. A trained, managed and stable workforce, with access to specialists as needed. Tooling and quality systems. Method: how to pilot a guideline, measure agreement, route by difficulty and counter automation bias. Capacity that scales up and down.

What distinguishes this from outsourcing is the interface. Annotators’ questions and disagreements reach the client’s experts and change the guidelines. The partner’s team stays on the account long enough to learn the domain. Quality is measured jointly against a reference both sides trust.

Safeguards worth insisting on

The Scale episode suggests several protections that have become more important.

Independence. Who owns the supplier, and who else does it serve? A supplier controlled by a competitor, or by a model developer with its own interests, sees information it may have reason to use.

Data rights. The contract should state that the client owns the data, the labels and the guidelines, that none of it is used to train the supplier’s or anyone else’s models, and that it is deleted or returned at the end.

Workforce stability and separation. Named, vetted teams, not an anonymous crowd. Separation from teams serving competitors. Controls on the use of external AI tools.

Security. Access controls, secure environments for sensitive data, and audit logs. For evaluation data, protection against leakage into any training set.

Portability. Guidelines, gold sets and annotator performance data should be the client’s and in a form that could be taken elsewhere. This is the defence against lock-in.

Transparency of process. The right to see who the annotators are in aggregate, how they were trained, how agreement and accuracy are measured, and how much of the work is automated.

Deciding for a given workload

A short set of questions places most data work.

How specific is the knowledge required? If any competent annotator could do it from a written guideline, it is generic. If it needs someone who knows your products, customers or standards, it is specific.

How would you know if it were done badly? If sampling and checking would reveal it, verification is easy. If it would surface only in model behaviour months later, it is hard.

How sensitive is it? Consider what the data, and the task itself, reveal about your strategy.

How steady is the demand? Continuous work can justify a standing team. Bursts favour external capacity.

Generic, verifiable, non-sensitive work should be bought on price and automated where possible. Specific, hard-to-verify, sensitive work should be kept close, with internal experts in control and a partner providing capacity and method under strong safeguards. Most organisations will have some of each and should manage them differently.

What this changes