Query unmapped fields without reindexing

ES|QL can query fields that are not defined in your index mapping, with no reindex and no change to the mapping. The SET unmapped_fields directive controls how each query handles them.

An unmapped field is a field in indexed documents that the index' mapping does not define. By default, ES|QL treats such fields as null and returns an error if a referenced field is not mapped in any index.

For more about how NULL behaves in expressions, filters, sorting, and aggregation, refer to Working with NULL values in ES|QL.

Without this capability, the usual fix is to add the field to your mapping and reindex your data before you can query it. On a large dataset, that reindex can take hours.

Querying unmapped fields helps in several common situations where the mapping does not yet match the data you want to explore:

• Explore new fields before mapping them: query the values of a field as soon as it appears, whether it arrives unexpectedly from an integration or an upstream change, then decide later whether the performance gains of formal mapping justify a reindex.
• Read the real values of a partially mapped field: when a field is mapped in some indices but not others, get its values everywhere in a single query, including the indices where it is unmapped.
• Run reusable queries across datasets: let a query continue when a field is not mapped in the data it runs against. This is useful for shared or saved queries and for data streams whose mappings change between rollovers. You can either ignore the missing data by returning null or try to load its values from _source.

The unmapped_fields setting accepts the following values, which range from strict to permissive. These values also control how ES|QL resolves a partially unmapped field, one that exists in the mapping of some indices but not others, across a multi-index query.

For the full syntax, refer to the SET unmapped_fields reference.

With LOAD, ES|QL reads the unmapped fields that your query names from _source and returns them as keyword columns. A named field that is absent from _source returns null.

SET unmapped_fields="load";
FROM partial_mapping_sample_data
| KEEP @timestamp, message, unmapped_message
| SORT @timestamp DESC
		

message is mapped and unmapped_message isn't, but both are returned as regular columns.

For a partially mapped field, one that is mapped in some queried indices but not others, LOAD reads values from _source in the indices where it's unmapped, even if the query doesn't name the field. For example, in FROM idx1, idx2, a field mapped only in idx1 is loaded from idx2.

If the field's mapped type isn't keyword, LOAD converts the loaded values to that type where possible.

LOAD only loads the unmapped fields that your query names. With LOAD_ALL, ES|QL also reads the rest of the unmapped fields from _source, so you can see what is in your documents without listing every field.

SET unmapped_fields="LOAD_ALL";
FROM partial_mapping_sample_data
| SORT @timestamp
| LIMIT 1
		

Mapped fields are returned as columns as usual. Every unmapped field is returned in the extra _unmapped_fields column, which has the type _source.

• One column, one JSON object per row. _unmapped_fields holds the unmapped fields of the row. It is null for a row that has none.
• Flattened, like a flattened field. Nested objects are flattened into dotted paths such as unmapped.nested, in the order the fields appear in _source. Values keep their JSON form from _source, so a field with several values is a JSON array. They become typed keyword columns only when the query names them.
• Empty values are left out. Explicit null values, empty arrays, and empty objects don't appear in the column.
• Fields you name become their own column and are no longer part of _unmapped_fields. A field that the query refers to, for example in WHERE, KEEP, or EVAL, is loaded the same way as with LOAD, as a keyword column. The same applies if a command creates a column with the same name as an unmapped field, for example with EVAL or RENAME.
  • A named field is always a keyword column, even when its value in _source is a number: 1 in _unmapped_fields becomes "1", and an array of numbers becomes a multivalued keyword. To get the original type back, convert it, for example with TO_LONG(field) or field::long.
• KEEP and DROP with wildcards apply to the unmapped fields. DROP unmapped_* removes the matching fields from _unmapped_fields. KEEP @timestamp, unmapped_* keeps only the matching fields in _unmapped_fields. A KEEP with only exact names doesn't return _unmapped_fields at all.
• Commands and functions that take field patterns don't see them. For example, full-text search functions such as KQL and QSTR don't search the fields in _unmapped_fields.
• Partially mapped fields behave as in LOAD. Where a field is mapped in some of the queried indices only, ES|QL reads it from the mapping where it exists and from _source where it doesn't. These fields are columns, not part of _unmapped_fields.

LOAD_ALL is a technical preview. For what it doesn't support yet, and what it costs, refer to Limitations specific to LOAD_ALL.

LOAD reads values from the stored _source. That is slower than querying a mapped field, because the values aren't stored in a data structure optimized for fast access. The trade-off is that the data is available immediately without a full reindex.

Latency is especially high when a filter or sort on an unmapped field cannot be pushed down to Lucene and forces a full scan of the index. For example:

SET unmapped_fields="load";
FROM idx
| SORT unmapped
| LIMIT 10
		

SET unmapped_fields="load";
FROM idx
| WHERE unmapped == "foo"
| LIMIT 10
		

In both cases, ES|QL cannot use a Lucene index on unmapped, so it must read documents (and parse _source for the loaded field) across the queried data. Large _source documents make this even slower. Prefer mapped fields for selective filters and sorts on large indices; use LOAD for exploration and for fields you haven't mapped yet. LOAD_ALL reads every unmapped field, so it is even more expensive.

LOAD does not actually load every field from _source. It loads:

• Partially mapped fields: ES|QL loads them from _source in the indices where they are unmapped, even when the query never mentions them. Refer to Load specific unmapped fields.
• Fully unmapped fields: fields that are not mapped in any queried index are loaded only when the query references them.

LOAD_ALL also loads the fully unmapped fields that the query doesn't reference. It returns them in the _unmapped_fields column.

LOAD and LOAD_ALL read a field from _source only while the index doesn't map it. Once you add the field to the mapping, ES|QL reads it like any other mapped field, from the data structures that were built when each document was indexed. Documents indexed before the mapping update have no such data for the field, so it is null for them, even though _source still contains the value. This is the same behavior as for any mapped field that you add to an existing index, and it applies with or without SET unmapped_fields.

For example, with "dynamic": false, a field f that is first read with LOAD returns its values for every document. After you map f as keyword, the same query returns null for the documents that were indexed before the mapping, and the real value only for newer documents. If the query uses LOAD_ALL, the old value is also missing from the _unmapped_fields column, because f is no longer an unmapped field.

The value is not lost. It remains in _source, and the get API still returns it. To make the older documents readable through the mapped field, rewrite them, for example with the update by query API or by reindexing into an index with the new mapping.

LOAD and LOAD_ALL do not support every command, function, and field type. Notable restrictions include:

• PROMQL is not supported with LOAD or LOAD_ALL.
• Referencing subfields of flattened parents is not supported.
• Partially mapped fields whose type has no implicit conversion from keyword (for example text or aggregate_metric_double) keep their mapped type but are not loaded from _source (i.e., they're set as null where unmapped). Using explicit cast, e.g., to_text allows loading from _source.

For the full list of restrictions, refer to the SET unmapped_fields reference.

Search functions can search a loaded field:

• MATCH
• MATCH_PHRASE

Because the loaded values have no index behind them, this is a runtime search: it scans values row by row, and analyzes text values with the standard analyzer rather than the analyzer in the mapping, where the field is mapped.

In addition to the restrictions above, LOAD_ALL has these limitations:

• It supports most commands, and a query that uses another command fails with an error that lists the supported ones. Notably, the TS source command is not supported.
• It doesn't apply inside views. The setting stops at a view's boundary: a view doesn't return an _unmapped_fields column, and the query returns null with a warning for an unmapped field that the view doesn't output. To load such fields, the view definition must itself SET unmapped_fields="load" and reference them. A view definition can't use LOAD_ALL. Indices and subqueries next to a view are loaded as usual.
• Reading every unmapped field uses more memory than LOAD, especially on documents with many fields or large arrays. If you know which fields you need, name them and use LOAD.

Unmapped fields differ from runtime fields. A runtime field is a computed field that can be defined in the index mapping, while an unmapped field is not in the mapping at all but can be present in documents. If a runtime field is part of the index mapping, ES|QL treats it like any other mapped field. In ES|QL, you create computed columns with the EVAL command. To learn more, refer to runtime fields.

To go deeper on unmapped fields and related capabilities, refer to these pages:

• SET unmapped_fields: the directive syntax, accepted values, and LOAD limitations.
• JSON_EXTRACT: extract values from JSON strings and _source.
• Retrieve unmapped fields: the equivalent include_unmapped option in the search fields API.
• ES|QL unmapped fields on Elastic Search Labs: the blog post that introduces the feature.