Index

Datatailr - Python SDK

The datatailr package provides a Python SDK for interacting with the Datatailr platform.

For a quickstart guide please launch the default workspace on your installation and open the README file.

Datatailr provides a simple interface for scheduling and managing data tasks, handling user access control, working with data blobs and secrets, and more.

App

Represents a web application or a dashboard deployment on Datatailr. This could be a Streamlit app, Dash app, or any other web-based data application. The implementation of the app does not need to follow any specific framework, as long as it can be started in the standard way (e.g., streamlit run app.py).

Example:

# app.py
import streamlit as st

def main():
    st.title("Hello Datatailr App")

if __name__ == "__main__":
    main()

The app can be tested locally by running:

streamlit run app.py

To deploy the app to Datatailr, you would create an App job in your Python code:

from app import main
from datatailr import App

app = App(
    name="Simple Dashboard App",
    entrypoint=main,
    python_requirements="streamlit")

app.run()

This will package and deploy the app to the Datatailr platform. The app definition and deployment status can be viewed at:

https://YOUR-DOMAIN.datatailr.com/job-details/simple-dashboard-app/1/dev/definition

The deployed app can be accessed by all users with the appropriate permissions from the app launcher page or directly via its URL:

https://YOUR-DOMAIN.datatailr.com/job/dev/simple-dashboard-app/

__init__(name, entrypoint, environment=Environment.DEV, image=None, run_as=None, resources=Resources(), acl=None, framework=None, python_version='3.12', python_requirements='', build_script_pre='', build_script_post='', env_vars={}, update_existing=False)

Initialize an App deployment.

Parameters:

Name	Type	Description	Default
name	str	Display name for the app.	required
entrypoint	Callable	The callable (function) that starts the application.	required
environment	Optional[Environment]	Target environment for the deployment.	DEV
image	Optional[Image]	Pre-configured container Image. When None, an image is built from python_version, python_requirements, and the build scripts.	None
run_as	Optional[Union[str, User]]	User or username to run the app as. Defaults to the currently signed-in user.	None
resources	Resources	CPU and memory resources for the container.	Resources()
acl	Optional[ACL]	Access control list. Defaults to standard permissions for the current user.	None
framework	Optional[str]	Optional framework name (e.g., 'streamlit', 'dash', 'flask'). If not provided, the framework is inferred from the entrypoint.	None
python_version	str	Python version for the container image.	'3.12'
python_requirements	str | List[str]	Python dependencies (see Image).	''
build_script_pre	str	Dockerfile commands to run before installing requirements.	''
build_script_post	str	Dockerfile commands to run after installing requirements.	''
env_vars	Dict[str, str | int | float | bool]	Environment variables passed to the running container.	{}
update_existing	bool	If True, load and update an existing job definition with the same name instead of creating a new one.	False

Blob

Interface for Datatailr's binary large object (blob) storage.

Blob storage allows you to persist and retrieve arbitrary files and binary data. Each blob is identified by a unique name (key).

Example

from datatailr import Blob
blob = Blob()
blob.put_blob("greeting", b"Hello, world!")
data = blob.get_blob("greeting")

delete(name)

Delete a blob.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to delete.	required

exists(name)

Check if a blob exists.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to check.	required

Returns:

Name	Type	Description
bool	bool	True if the blob exists, False otherwise.

get_blob(name)

Get a blob object.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to retrieve.	required

Returns:

Name	Type	Description
Blob	bytes	The blob object.

get_file(name, path)

Copy a blob file to a local file.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to retrieve.	required
path	str	The path to store the blob as a file.	required

ls(path)

List files in the specified path.

:param path: The path to list files from. :return: A list of file names in the specified path.

put_blob(name, blob)

Put a blob object into the blob storage.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to create.	required
blob	bytes | str	The blob object to store.	required

put_file(name, path)

Copy a local file to a blob.

Parameters:

Name	Type	Description	Default
name	str	The name of the blob to create.	required
path	str	The path of the local file to copy.	required

Environment

Enum representing different environments for DataTailr jobs.

ExcelAddin

Represents an Excel add-in deployment on Datatailr.

An Excel add-in exposes Python functions as Excel worksheet functions, allowing users to call server-side computations directly from Excel spreadsheets.

Example

from datatailr import ExcelAddin

def price_option(spot, strike, vol, rate, expiry):
    # Black-Scholes pricing logic
    ...

addin = ExcelAddin(
    name="Options Pricer",
    entrypoint=price_option,
    python_requirements="numpy,scipy",
)
addin.run()

__init__(name, entrypoint, environment=Environment.DEV, image=None, run_as=None, resources=Resources(), acl=None, python_version='3.12', python_requirements='', build_script_pre='', build_script_post='', env_vars={}, update_existing=False)

Initialize an Excel add-in deployment.

Parameters:

Name	Type	Description	Default
name	str	Display name for the add-in.	required
entrypoint	Callable	The callable (function) to expose as an Excel function.	required
environment	Optional[Environment]	Target environment for the deployment.	DEV
image	Optional[Image]	Pre-configured container Image.	None
run_as	Optional[Union[str, User]]	User or username to run the add-in as.	None
resources	Resources	CPU and memory resources for the container.	Resources()
acl	Optional[ACL]	Access control list.	None
python_version	str	Python version for the container image.	'3.12'
python_requirements	str | List[str]	Python dependencies (see Image).	''
build_script_pre	str	Dockerfile commands to run before installing requirements.	''
build_script_post	str	Dockerfile commands to run after installing requirements.	''
env_vars	Dict[str, str | int | float | bool]	Environment variables passed to the running container.	{}
update_existing	bool	If True, update an existing job definition.	False

Group

Representing a Datatailr Group.

This class provides methods to interact with the Datatailr Group API. It allows you to create, update, delete, and manage groups within the Datatailr platform.

Attributes:

Name	Type	Description
name	str	The name of the group.
members	list	A list of members in the group.
group_id	int	The unique identifier for the group.

Static Methods

add(name: str) -> 'Group': Create a new group with the specified name. get(name: str) -> 'Group': Retrieve a group by its name. list() -> list: List all groups available in the Datatailr platform. remove(name: str) -> None: Remove a group by its name. exists(name: str) -> bool: Check if a group exists by its name.

Instance Methods

add_users(usernames: list) -> None: Add users to the group. remove_users(usernames: list) -> None: Remove users from the group.

group_id property

The unique numeric identifier of the group.

members property

List of user IDs that are members of this group.

name property

The name of the group.

__init__(name)

Create a Group instance by fetching group data from the platform.

Parameters:

Name	Type	Description	Default
name	str	The name of an existing group.	required

Raises:

Type	Description
ValueError	If a group with the given name does not exist.

add(name) staticmethod

Create a new group on the Datatailr platform.

Parameters:

Name	Type	Description	Default
name	str	The name for the new group.	required

Returns:

Type	Description
Optional[Group]	The newly created Group, or None if the operation failed.

add_users(usernames)

Add one or more users to this group.

Parameters:

Name	Type	Description	Default
usernames	list	A list of usernames to add.	required

Raises:

Type	Description
ValueError	If the group name is not set.

exists(name) staticmethod

Check whether a group with the given name exists.

Parameters:

Name	Type	Description	Default
name	str	The group name to look up.	required

Returns:

Type	Description
bool	True if the group exists, False otherwise.

from_dict(data) classmethod

Construct a Group instance from a dictionary.

Parameters:

Name	Type	Description	Default
data	dict	A dictionary with keys "name", "members", and "group_id".	required

Returns:

Type	Description
Group	A Group instance populated from the dictionary.

Raises:

Type	Description
ValueError	If required keys are missing from data.

get(name_or_id) staticmethod

Retrieve an existing group by name or numeric ID.

Parameters:

Name	Type	Description	Default
name_or_id	Union[str, int]	The group name (str) or group ID (int).	required

Returns:

Type	Description
Group	The matching Group instance.

Raises:

Type	Description
ValueError	If no group with the given name or ID exists.

is_member(user)

Check whether a user belongs to this group.

Parameters:

Name	Type	Description	Default
user		A User instance or a user ID (int).	required

Returns:

Type	Description
bool	True if the user is a member, False otherwise.

Raises:

Type	Description
ValueError	If the group name is not set.

ls() staticmethod

List all groups on the Datatailr platform.

Returns:

Type	Description
list	A list of Group instances.

remove(name) staticmethod

Remove (delete) a group from the platform.

Parameters:

Name	Type	Description	Default
name	str	The name of the group to remove.	required

remove_users(usernames)

Remove one or more users from this group.

Parameters:

Name	Type	Description	Default
usernames	list	A list of usernames to remove.	required

Raises:

Type	Description
ValueError	If the group name is not set.

Image

Represents a container image for a job. The image is defined based on its' python dependencies and two 'build scripts' expressed as Dockerfile commands. All attributes can be initialized with either a string or a file name.

__init__(python_version='auto', python_requirements='', build_script_pre='', build_script_post='', branch_name=None, commit_hash=None, path_to_repo=None, path_to_module=None)

Initialize an Image configuration.

Parameters:

Name	Type	Description	Default
python_version	str	Python version for the container (e.g. "3.12"). Use "auto" to match the current interpreter version.	'auto'
python_requirements	str | list[str]	Python dependencies. Accepts a pip-freeze string, a list of package specifiers, a path to a requirements.txt file, or "auto" to run pip freeze.	''
build_script_pre	str	Dockerfile commands to execute before installing Python requirements. Can be a string or a file path.	''
build_script_post	str	Dockerfile commands to execute after installing Python requirements. Can be a string or a file path.	''
branch_name	Optional[str]	Git branch name (populated automatically during build).	None
commit_hash	Optional[str]	Git commit hash (populated automatically during build).	None
path_to_repo	Optional[str]	Absolute path to the source repository root.	None
path_to_module	Optional[str]	Relative path from the repository root to the Python module containing the entrypoint.	None

from_dict(data)

Create an Image instance from a dictionary representation.

to_dict()

Convert the Image instance to a dictionary representation.

to_json()

Convert the Image instance to a JSON string representation.

update(**kwargs)

Update one or more Image attributes.

Parameters:

Name	Type	Description	Default
**kwargs		Attribute names and their new values.	{}

Raises:

Type	Description
TypeError	If a string-only attribute receives a non-string value.
AttributeError	If an attribute name does not exist on the Image.

KV

A distributed key-value store for storing and retrieving configuration data, application settings, and other key-value pairs.

Note that different environments (dev/pre/prod) have separate key-value stores.

Each key-value pair can have its own access control list (ACL) to manage permissions for reading and writing.

Example:

from datatailr import KV, ACL, User, Group, Permission
kv = KV()
kv.put("db_url", "postgresql://user:password@host:port/dbname")
acls = ACL({Permission.READ: [User.get("user1"), Group.get("group1")],
            Permission.WRITE: [User.get("user2")]})
kv.add_acl("db_url", acls)
db_url = kv.get("db_url")
all_keys = kv.ls()
kv.delete("db_url")

acls(key)

Retrieve the access control list (ACL) for a specific key in the key-value store.

Parameters:

Name	Type	Description	Default
key	str	The key for which to retrieve the ACL.	required

Returns:

Type	Description
ACL	dict[str, list[str]]: A dictionary containing lists of users/groups with 'read' and 'write' permissions.

Example:

from datatailr import KV
kv = KV()
acl = kv.acls("db_url")

add_acl(key, acls)

Add an entry to the access control list (ACL) for a specific key in the key-value store.

Parameters:

Name	Type	Description	Default
key	str	The key for which to add the ACL entry.	required
entity	str	The user or group to which the permission is granted.	required
permission	str	The permission to grant ('read' or 'write').	required

Returns:

Type	Description
None	None

Example:

from datatailr import KV, ACL
kv = KV()
acls = ACL(read=["user1", "group1"], write=["user2"])
kv.add_acl("db_url", acls)

delete(key)

Delete a key-value pair from the key-value store by its key.

Parameters:

Name	Type	Description	Default
key	str	The key of the value to delete.	required

Returns: None Example:

from datatailr import KV
kv = KV()
kv.delete("db_url")  # Deletes the key-value pair with key "db_url"

get(key)

Retrieve a value by its key from the key-value store. Note that different environments (dev/pre/prod) have separate key-value stores.

Parameters:

Name	Type	Description	Default
key	str	The key of the value to retrieve.	required

Returns:

Name	Type	Description
str	str	The value associated with the key.

Example:

from datatailr import KV
kv = KV()
db_url = kv.get("db_url")

ls()

List all keys stored in the key-value store for the current environment (dev/pre/prod).

Returns:

Type	Description
list[dict[str, str | ACL]]	list[dict[str, str|ACL]]: A list of all keys in the key-value store along with their ACLs.

Example:

from datatailr import KV
kv = KV()
all_keys = kv.ls()

put(key, value, acl=None)

Store a key-value pair in the key-value store. Note that different environments (dev/pre/prod) have separate key-value stores.

Parameters:

Name	Type	Description	Default
key	str	The key under which to store the value.	required
value	str	The value to store.	required
acl	ACL	The access control list for the key-value pair.	None

Returns:

Type	Description
None	None

Example:

from datatailr import KV, ACL, Permission, User, Group
kv = KV()
acls = ACL({Permission.READ: [User.get("user1"), Group.get("group1")],
            Permission.WRITE: [User.get("user2")]})
kv.put("db_url", "postgresql://user:password@host:port/dbname", acls)

set_acls(key, acls)

Set the access control list (ACL) for a specific key in the key-value store, replacing any existing ACL entries.

Parameters:

Name	Type	Description	Default
key	str	The key for which to set the ACL.	required
entity	str	The user or group for which to set the permission.	required
permission	str	The permission to set ('read' or 'write').	required

Returns:

Type	Description
None	None

Example:

from datatailr import KV, ACL
kv = KV()
acls = ACL(read=["user1", "group1"], write=["user2"])
kv.set_acls("db_url", acls)

Permission

Enumeration of permission types used in Access Control Lists.

Each permission controls a different aspect of access to a Datatailr resource.

Attributes:

Name	Type	Description
READ		Allows reading or viewing the resource.
WRITE		Allows modifying or updating the resource.
OPERATE		Allows executing or operating the resource (e.g., running a job).
ACCESS		Allows accessing the resource (e.g., opening an app).
PROMOTE		Allows promoting the resource to the next environment.

Resources dataclass

Represents the compute resources allocated to a job container.

Attributes:

Name	Type	Description
memory	str	Memory limit as a string (e.g. "256m", "1g").
cpu	float	Number of CPU cores to allocate (e.g. 1, 0.5).

Schedule

Represents a schedule for batch jobs.

__init__(cron_expression='', at_minutes=None, every_minute=None, at_hours=None, every_hour=None, weekdays=None, day_of_month=None, in_month=None, every_month=None, timezone=None, run_after_job_uuid=None, run_after_job_name=None, run_after_job_condition=None)

Initialize a Schedule.

You can either provide a raw cron_expression or use the human-readable helper parameters (which are compiled into cron syntax automatically).

Parameters:

Name	Type	Description	Default
cron_expression	str	A raw cron string (e.g. "0 */2 * * *"). If provided alongside helper parameters, the helpers take precedence.	''
at_minutes	list[int] | None	Specific minutes past the hour to run (e.g. [0, 30] for ":00" and ":30").	None
every_minute	int | None	Run every N minutes (e.g. 5 for every 5 minutes).	None
at_hours	list[int] | None	Specific hours of the day to run (0 -- 23).	None
every_hour	int | None	Run every N hours.	None
weekdays	list[str] | None	Days of the week to run (e.g. ["mon", "wed", "fri"]).	None
day_of_month	int | None	Day of the month to run (1 -- 31).	None
in_month	list[str] | None	Months to run in (e.g. ["jan", "apr", "jul", "oct"]).	None
every_month	int | None	Run every N months.	None
timezone	str | None	IANA timezone name (e.g. "America/New_York").	None
run_after_job_uuid	str | None	UUID of a job that must complete before this schedule triggers.	None
run_after_job_name	str | None	Name of a job that must complete before this schedule triggers.	None
run_after_job_condition	str | None	Required completion status of the predecessor job (e.g. "success").	None

Example

from datatailr import Schedule
# Every weekday at 08:00 and 16:00 UTC
s = Schedule(at_hours=[8, 16], weekdays=["mon","tue","wed","thu","fri"])

get_cron_string()

Returns the compiled cron string.

Secrets

Interface for managing secrets stored in the Datatailr platform.

Secrets are sensitive key-value pairs (passwords, API keys, tokens, etc.) that are stored securely and scoped to the current environment (dev/pre/prod). Access is controlled by per-secret ACLs.

Example

from datatailr import Secrets
secrets = Secrets()
db_password = secrets.get("database_password")
all_secrets = secrets.ls()

get(key)

Retrieve a secret value by its key. Only secrets that are available at the current environment (dev/pre/prod) can be accessed. The user must have the appropriate permissions to access the secret.

Parameters:

Name	Type	Description	Default
key	str	The key of the secret to retrieve.	required

Returns:

Name	Type	Description
str	str	The value of the secret.

Example:

from datatailr import Secrets
secrets = Secrets()
db_password = secrets.get("database_password")

ls()

List all available secret keys in the current environment (dev/pre/prod). The user must have the appropriate permissions to list the secrets.

Returns:

Type	Description
list[dict[str, str | ACL]]	list[dict[str, str|ACL]]: A list of all secret keys along with their ACLs.

Example:

from datatailr import Secrets
secrets = Secrets()
all_secrets = secrets.ls()

Service

Represents a long-running background service deployment on Datatailr.

A service runs continuously (e.g., an API server, a message consumer, or any always-on process). It is restarted automatically if it exits.

Example

from datatailr import Service

def run_server():
    import uvicorn
    uvicorn.run("myapi:app", host="0.0.0.0", port=8000)

svc = Service(
    name="My API Service",
    entrypoint=run_server,
    python_requirements="fastapi,uvicorn",
)
svc.run()

__init__(name, entrypoint, environment=Environment.DEV, image=None, run_as=None, resources=Resources(), acl=None, python_version='3.12', python_requirements='', build_script_pre='', build_script_post='', env_vars={}, update_existing=False)

Initialize a Service deployment.

Parameters:

Name	Type	Description	Default
name	str	Display name for the service.	required
entrypoint	Callable	The callable (function) that starts the service.	required
environment	Optional[Environment]	Target environment for the deployment.	DEV
image	Optional[Image]	Pre-configured container Image.	None
run_as	Optional[Union[str, User]]	User or username to run the service as.	None
resources	Resources	CPU and memory resources for the container.	Resources()
acl	Optional[ACL]	Access control list.	None
python_version	str	Python version for the container image.	'3.12'
python_requirements	str | List[str]	Python dependencies (see Image).	''
build_script_pre	str	Dockerfile commands to run before installing requirements.	''
build_script_post	str	Dockerfile commands to run after installing requirements.	''
env_vars	Dict[str, str | int | float | bool]	Environment variables passed to the running container.	{}
update_existing	bool	If True, update an existing job definition.	False

User

Representing a Datatailr User.

This class provides methods to interact with the Datatailr User API. It allows you to create, update, delete, and manage users within the Datatailr platform.

Attributes:

Name	Type	Description
first_name	str	The first name of the user.
last_name	str	The last name of the user.
name	str	The username of the user.
email	str	The email address of the user.
user_id	int	The unique identifier for the user.
groups	list	List of groups the user belongs to.
is_system_user	bool	Indicates if the user is a system user.

Static Methods

signed_user() -> Optional['User']: Retrieve the currently signed-in user, if available. add(name: str, first_name: str = None, last_name: str = None, email: str = None, password: str = None, groups: list[str] = [], is_system_user: bool = False) -> 'User': Create a new user with the specified username, first name, last name, and email. get(name: str) -> 'User': Retrieve a user by their username. exists(name: str) -> bool: Check if a user exists by their username. ls() -> list: List all users available in the Datatailr platform. remove(name: str) -> None: Remove a user by their username.

Instance Methods

verify() -> None: Refresh the user information from the Datatailr API.

email property

The email address of the user.

first_name property

The first name of the user.

groups property

List of group names the user belongs to.

is_system_user property

Whether this user is a system (non-interactive) user.

last_name property

The last name of the user.

name property

The username.

user_id property

The unique numeric identifier of the user.

__init__(name=None, id=None)

Create a User instance by fetching user data from the platform.

At least one of name or id must be provided. When both are given, name takes precedence.

Parameters:

Name	Type	Description	Default
name	str | None	The username to look up.	None
id	int | None	The numeric user ID to look up.	None

Raises:

Type	Description
ValueError	If neither name nor id is provided.

add(name, first_name, last_name, email, password, groups=None, is_system_user=False) staticmethod

Create a new user on the Datatailr platform.

Parameters:

Name	Type	Description	Default
name	str	The username for the new user.	required
first_name	str	The user's first name.	required
last_name	str	The user's last name.	required
email	str	The user's email address.	required
password	str	The login password. Ignored for system users.	required
groups	list[str] | None	Optional list of group names to add the user to.	None
is_system_user	bool	If True, create a non-interactive system user.	False

Returns:

Type	Description
Optional['User']	The newly created User, or None if the operation failed.

exists(name) staticmethod

Check whether a user with the given username exists.

Parameters:

Name	Type	Description	Default
name	str	The username to look up.	required

Returns:

Type	Description
bool	True if the user exists, False otherwise.

get(name_or_id) staticmethod

Retrieve an existing user by username or numeric ID.

Parameters:

Name	Type	Description	Default
name_or_id	str | int	The username (str) or user ID (int).	required

Returns:

Type	Description
User	The matching User instance.

Raises:

Type	Description
ValueError	If no user with the given name or ID exists.

ls() staticmethod

from datatailr import User User.ls() [User(name=root, first_name=Super User, last_name=, email=None, user_id=0, groups=[], is_system_user=True), ...]

remove(name) staticmethod

Remove (delete) a user from the platform.

Parameters:

Name	Type	Description	Default
name	str	The username of the user to remove.	required

signed_user() staticmethod

Retrieve the currently signed-in (authenticated) user.

Returns:

Type	Description
User	The User instance of the currently authenticated user.

Raises:

Type	Description
PermissionError	If no signed-in user is found.

verify()

Verify the user's authentication signature with the platform.

This refreshes and validates the user's credentials against the Datatailr authentication service.

is_dt_installed()

Check if the native Datatailr CLI is present in /opt/datatailr/bin/dt.

task(memory=DEFAULT_TASK_MEMORY, cpu=DEFAULT_TASK_CPU)

Decorator that marks a function as a workflow task (batch job).

Use this via the public alias @task() to declare individual computational steps inside a @workflow-decorated function.

Parameters:

Name	Type	Description	Default
memory	str	Memory limit for this task's container (e.g. "256m", "1g").	DEFAULT_TASK_MEMORY
cpu	float	Number of CPU cores to allocate for this task (e.g. 1, 0.5).	DEFAULT_TASK_CPU

Returns:

Type	Description
	A decorator that wraps the target function so it can participate
	in DAG-based workflow orchestration.

Example

from datatailr import task

@task(memory="512m", cpu=2)
def heavy_computation(x, y):
    return x ** y