Sandboxes

Kusto can run sandboxes for specific flows that must be run in a secure and isolated environment. Examples of these flows are user-defined scripts that run using the Python plugin or the R plugin.

Sandboxes are run locally (meaning, processing is done close to the data), with no extra latency for remote calls.

Prerequisites and limitations

• Sandboxes that run on VM sizes supporting nested virtualization are implemented using Hyper-V technology and have no limitations.
• Sandboxes that run on VM sizes not supporting nested virtualization are implemented using a proprietary legacy technology and are subject to some limitations.
• The image for running the sandboxes is deployed to every cluster node and requires dedicated SSD space to run.
  • The estimated size is between 10-20 GB.
  • This affects the cluster's data capacity, and may affect the cost of the cluster.

Runtime

• A sandboxed query operator may use one or more sandboxes for its execution.
  • A sandbox is only used for a single query and is disposed of once that query completes.
  • When a node is restarted, for example, as part of a service upgrade, all running sandboxes on it are disposed of.
• Each node maintains a predefined number of sandboxes that are ready for running incoming requests.
  • Once a sandbox is used, a new one is automatically made available to replace it.
• If there are no pre-allocated sandboxes available to serve a query operator, it will be throttled until new sandboxes are available. For more information, see Errors. New sandbox allocation could take up to 10-15 seconds per sandbox, depending on the SKU and available resources on the data node.

Sandbox parameters

Some of the parameters can be controlled using a cluster-level sandbox policy, for each kind of sandbox.

• Number of sandboxes per node: The number of sandboxes per node is limited.
  • Requests that are made when there's no available sandbox will be throttled.
• Initialize on startup: if set to false (default), sandboxes are lazily initialized on a node, the first time a query requires a sandbox for its execution. Otherwise, if set to true, sandboxes are initialized as part of service startup.
  • This means that the first execution of a plugin that uses sandboxes on a node will include a short warm-up period.
• CPU: The maximum rate of CPU a sandbox can consume of its host's processors is limited (default is 50%).
  • When the limit is reached, the sandbox's CPU use is throttled, but execution continues.
• Memory: The maximum amount of RAM a sandbox can consume of its host's RAM is limited.
  • Default memory for Hyper-V technology is 1 GB, and for legacy sandboxes 20 GB.
  • Reaching the limit results in termination of the sandbox, and a query execution error.

Sandbox limitations

• Network: A sandbox can't interact with any resource on the virtual machine (VM) or outside of it.
  • A sandbox can't interact with another sandbox.

Errors

VM Sizes supporting nested virtualization

The following table lists all modern VM sizes that support Hyper-V sandbox technology.