A COPY INTO option for CSV sources that states whether the files use UTF8, the default, or UTF16 character encoding.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains ENCODING in context, with comparison tables and the common traps.
Terms in this definition
- COPY INTO
Loads files from a volume or cloud storage into a Delta table that already exists, skipping anything loaded before, so reruns are safe. Good for thousands of files; for millions, Auto Loader scales further.
- Cluster Shared Volumes
Shared failover cluster storage that all nodes write to simultaneously, typically for Scale-Out File Server or Hyper-V. On SAN-backed storage, ReFS-formatted volumes work in redirected mode.
Related terms
- ASCII
An encoding for human-readable files like JSON or CSV that turns their bytes into printable characters. Unicode supports far more characters.
- BOM
Bytes placed first in a text file to signal its encoding. Fine-tuning data should be saved as UTF-8 with a BOM, keeping each file below 512 MB; ASCII, UTF-16 or ISO-8859-1 files risk failing validation.
- Embeddings
Vectors of numbers encoding what text means, placing related content close together as judged by cosine similarity. Switching model or vector size requires embedding all content again.
- Prompt Shields for user prompts
The Prompt Shields check for direct attacks within the user's input, such as attempts to change rules, role-play personas, mocked-up conversations and encoding tricks. Documents and images are not scanned.
- UTF-8
The text encoding required for Intune remediation scripts. If the file has a byte order mark and signature checks are enforced, the script fails.
- V-Order
An optimisation Fabric applies to Parquet files as they are written, covering sorting, row-group layout, encoding and compression, so reads (above all Direct Lake) are quicker at some cost to writes. New workspaces have it switched off, and
OPTIMIZE ... VORDERapplies it to files already written. - Value encoding
VertiPaq applies it to numbers, keeping the actual values and achieving its strongest compression. Columns holding text use hash encoding instead, built on a dictionary of distinct entries, which is why turning a text key with many unique values into whole numbers may reduce model size.