API reference

sai_airflow_plugins.hooks

class sai_airflow_plugins.hooks.fabric_hook.FabricHook(ssh_conn_id=None, inline_ssh_env=False, *args, **kwargs)[source]

Bases: SSHHook

This hook allows you to connect to an SSH remote host and run commands on it using the [Fabric](https://www.fabfile.org) library. It inherits from SSHHook and uses its input arguments for setting up the connection.

Parameters:

  • ssh_conn_id (str) – connection id from airflow Connections

  • inline_ssh_env (bool) – whether to send environment variables “inline” as prefixes in front of command strings (export VARNAME=value && mycommand here), instead of trying to submit them through the SSH protocol itself (which is the default behavior). This is necessary if the remote server has a restricted AcceptEnv setting (which is the common default).

get_fabric_conn()[source]

Creates a Fabric Connection object using the settings in this hook.

Return type:

Connection

Returns:

Connection object

get_generic_pass_responder()[source]

Creates a responder for a generic password prompt. It replies with the password of the SSH connection. This is useful if you execute other ssh commands on the remote host, for example scp and rsync. Note: only use this if the SSH connection is configured with a password and not a key.

Return type:

FailingResponder

Returns:

FailingResponder object; raises AirflowException if no password was configured in the connection

get_sudo_pass_responder()[source]

Creates a responder for the sudo password prompt. It replies with the password of the SSH connection. Note: only use this if the SSH connection is configured with a password and not a key.

Return type:

FailingResponder

Returns:

FailingResponder object; raises AirflowException if no password was configured in the connection

static get_unknown_host_key_responder()[source]

Creates a responder for a host authenticity check with an unknown key. It replies yes to continue connecting. This is useful if you execute other ssh commands on the remote host, for example scp and rsync.

Return type:

FailingResponder

Returns:

FailingResponder object

class sai_airflow_plugins.hooks.mattermost_webhook_hook.MattermostWebhookHook(http_conn_id=None, webhook_token=None, message='', attachments=None, props=None, post_type=None, channel=None, username=None, icon_emoji=None, icon_url=None, proxy=None, extra_options=None, *args, **kwargs)[source]

Bases: HttpHook

This hook allows you to post messages to Mattermost using incoming webhooks. It takes either a Mattermost webhook token directly or a connection that has a Mattermost webhook token. If both are supplied, http_conn_id will be used as base_url, and webhook_token will be taken as endpoint, the relative path of the url.

Each Mattermost webhook token can be pre-configured to use a specific channel, username and icon. You can override these defaults in this hook.

This hook is based on airflow.contrib.hooks.SlackWebhookHook as the Mattermost interface is largely similar to that of Slack.

Parameters:

  • http_conn_id (Optional[str]) – connection that optionally has a Mattermost webhook token in the extra field

  • webhook_token (Optional[str]) – Mattermost webhook token. If http_conn_id isn’t supplied this should be the full webhook url.

  • message (str) – The message you want to send on Mattermost

  • attachments (Optional[List[Dict[str, Any]]]) – The attachments to send on Mattermost. Should be a list of dictionaries representing Mattermost attachments.

  • props (Optional[Dict[str, Any]]) – The props to send on Mattermost. Should be a dictionary representing Mattermost props.

  • post_type (Optional[str]) – Sets an optional Mattermost post type, mainly for use by plugins. If supplied, must begin with custom_

  • channel (Optional[str]) – The channel the message should be posted to

  • username (Optional[str]) – The username to post with

  • icon_emoji (Optional[str]) – The emoji to use as icon for the user posting to Mattermost

  • icon_url (Optional[str]) – The icon image URL string to use in place of the default icon.

  • proxy (Optional[str]) – Proxy to use to make the Mattermost webhook call

  • extra_options (Optional[Dict[str, Any]]) – Extra options for http hook

execute()[source]

Execute the Mattermost webhook call

sai_airflow_plugins.operators

class sai_airflow_plugins.operators.fabric_operator.FabricOperator(fabric_hook=None, ssh_conn_id=None, remote_host=None, command=None, use_sudo=False, use_sudo_shell=False, sudo_user=None, watchers=None, add_sudo_password_responder=False, add_generic_password_responder=False, add_unknown_host_key_responder=False, connect_timeout=10, environment=None, inline_ssh_env=False, xcom_push_key=None, strip_stdout=False, get_pty=False, keepalive=0, *args, **kwargs)[source]

Bases: BaseOperator

Operator to execute commands on a remote host using the [Fabric](https://www.fabfile.org) library. It uses FabricHook for the connection configuration, which is derived from the standard SSHHook.

The advantage of this operator over the standard SSHOperator is that you can add watchers that respond to specific command output. A number of predefined watchers are included in this operator. Note, however, that some of these require the FabricHook to be configured with a password and not a private key.

Parameters:

  • fabric_hook (Optional[FabricHook]) – predefined fabric_hook to use for remote execution. Either fabric_hook or ssh_conn_id needs to be provided.

  • ssh_conn_id (Optional[str]) – connection id from airflow Connections. ssh_conn_id will be ignored if fabric_hook is provided. (templated)

  • remote_host (Optional[str]) – remote host to connect. (templated) Nullable. If provided, it will replace the remote_host which was defined in fabric_hook or predefined in the connection of ssh_conn_id.

  • command (str) – command to execute on remote host. (templated)

  • use_sudo (Optional[bool]) – uses Fabric’s sudo function instead of run. Because this function automatically adds a responder for the password prompt, parameter add_sudo_password_responder will be ignored. It uses the SSH connection’s password as reply.

  • use_sudo_shell (Optional[bool]) – wraps the command in a sudo shell. The difference with use_sudo is that Fabric’s sudo function only executes a single command as sudo. Within a sudo shell a full script can be run, e.g. a templated script file. This does not automatically add a sudo responder for the password prompt. Use add_sudo_password_responder as necessary. This parameter can’t be used at the same time as use_sudo.

  • sudo_user (Optional[str]) – If use_sudo or use_sudo_shell is True, run the command as this user. If not supplied it will run as root. This parameter is ignored if use_sudo and use_sudo_shell are False.

  • watchers (Optional[List[Dict[str, Any]]]) – Watchers for responding to specific command output. This is a list of dicts specifying a class of type StreamWatcher and its arguments. For each dict the corresponding object is created and added to Fabric’s run function. It’s done this way because arguments to an operator are pickled and StreamWatcher objects are derived from thread.local which can’t be pickled. Example: >>> {“watchers”: [{“class”: Responder, “pattern”: r”Continue?”, “response”: “yesn”}]} See also: http://docs.pyinvoke.org/en/latest/concepts/watchers.html

  • add_sudo_password_responder (Optional[bool]) – adds a responder for the sudo password prompt. It replies with the password of the SSH connection. Set this to True if your command contains one or more sudo statements. If you use use_sudo instead, then don’t use this parameter.

  • add_generic_password_responder (Optional[bool]) – adds a responder for a generic password prompt. It replies with the password of the SSH connection. This is useful if you execute other ssh commands on the remote host, for example scp and rsync.

  • add_unknown_host_key_responder (Optional[bool]) – adds a responder for a host authenticity check with an unknown key. It replies yes to continue connecting. This is useful if you execute other ssh commands on the remote host, for example scp and rsync.

  • connect_timeout (Optional[int]) – Connection timeout, in seconds. The default is 10.

  • environment (Optional[Dict[str, Any]]) – a dict of shell environment variables. Note that the server will reject them silently if AcceptEnv is not set in SSH config. In such cases setting inline_ssh_env to True may help. (templated)

  • inline_ssh_env (Optional[bool]) – whether to send environment variables “inline” as prefixes in front of command strings (export VARNAME=value && mycommand here), instead of trying to submit them through the SSH protocol itself (which is the default behavior). This is necessary if the remote server has a restricted AcceptEnv setting (which is the common default).

  • xcom_push_key (Optional[str]) – push stdout to an XCom with this key. If None (default), or stdout is empty, then no XCom will be pushed.

  • strip_stdout (Optional[bool]) – strip leading and trailing whitespace from stdout. Useful, for example, when pushing stdout to an XCom and you don’t want a trailing newline in it.

  • get_pty (Optional[bool]) – request a pseudo-terminal from the server, instead of connecting directly to the stdout/stderr streams. This may be necessary when running programs that require a terminal. Note that stderr output will be included in stdout, and thus added to an XCom when using xcom_push_key.

  • keepalive (Optional[int]) – The number of seconds to send keepalive packets to the server. This corresponds to the ssh option ServerAliveInterval. The default is 0, which disables keepalive.

execute(context)[source]

Executes self.command over the configured SSH connection.

Parameters:

context (Dict) – Context dict provided by airflow

Returns:

True if the command executed correctly. On an error, raises AirflowException.

execute_fabric_command()[source]

Executes self.command over the configured SSH connection.

Return type:

Result

Returns:

The Result object from Fabric’s run method

template_ext: Sequence[str] = ('.sh',)
template_fields: Sequence[str] = ('ssh_conn_id', 'command', 'remote_host', 'environment')
ui_color: str = '#ebfaff'
class sai_airflow_plugins.operators.mattermost_webhook_operator.MattermostWebhookOperator(http_conn_id=None, webhook_token=None, message='', attachments=None, props=None, post_type=None, channel=None, username=None, icon_emoji=None, icon_url=None, proxy=None, extra_options=None, *args, **kwargs)[source]

Bases: SimpleHttpOperator

This operator allows you to post messages to Mattermost using incoming webhooks. It takes either a Mattermost webhook token directly or a connection that has a Mattermost webhook token. If both are supplied, http_conn_id will be used as base_url, and webhook_token will be taken as endpoint, the relative path of the url.

Each Mattermost webhook token can be pre-configured to use a specific channel, username and icon. You can override these defaults in this operator.

This operator is based on airflow.contrib.operators.SlackWebhookOperator as the Mattermost interface is largely similar to that of Slack.

Parameters:

  • http_conn_id (Optional[str]) – connection that optionally has a Mattermost webhook token in the extra field

  • webhook_token (Optional[str]) – Mattermost webhook token. If http_conn_id isn’t supplied this should be the full webhook url.

  • message (str) – The message you want to send on Mattermost

  • attachments (Optional[List[Dict[str, Any]]]) – The attachments to send on Mattermost. Should be a list of dictionaries representing Mattermost attachments.

  • props (Optional[Dict[str, Any]]) – The props to send on Mattermost. Should be a dictionary representing Mattermost props.

  • post_type (Optional[str]) – Sets an optional Mattermost post type, mainly for use by plugins. If supplied, must begin with custom_

  • channel (Optional[str]) – The channel the message should be posted to

  • username (Optional[str]) – The username to post with

  • icon_emoji (Optional[str]) – The emoji to use as icon for the user posting to Mattermost

  • icon_url (Optional[str]) – The icon image URL string to use in place of the default icon.

  • proxy (Optional[str]) – Proxy to use to make the Mattermost webhook call

  • extra_options (Optional[Dict[str, Any]]) – Extra options for http hook

execute(context)[source]

Call the MattermostWebhookHook to post the provided Mattermost message

template_fields: Sequence[str] = ['webhook_token', 'message', 'attachments', 'props', 'post_type', 'channel', 'username', 'proxy', 'extra_options']
class sai_airflow_plugins.operators.conditional_skip_mixin.ConditionalSkipMixin(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: object

Mixin for making operators and sensors conditional. If the condition evaluates to True the operator or sensor executes normally, otherwise it skips the task.

Note that you should correctly set the template_field in a derived class to include both the operator’s and this mixin’s templated fields. Example:

class MyConditionalOperator(ConditionalSkipMixin, MyOperator):

template_fields = MyOperator.template_fields + ConditionalSkipMixin.template_fields

Parameters:

  • condition_callable (Callable) – A callable that should evaluate to a truthy or falsy value to execute or skip the task respectively. Note that Airflow’s context is also passed as keyword arguments so you need to define **kwargs in your function header. (templated)

  • condition_kwargs (Optional[Dict]) – a dictionary of keyword arguments that will get unpacked in condition_callable. (templated)

  • condition_args (Optional[Iterable]) – a list of positional arguments that will get unpacked in condition_callable. (templated)

  • condition_provide_context (Optional[bool]) – if set to true, Airflow will pass a set of keyword arguments that can be used in your condition callable. This set of kwargs correspond exactly to what you can use in your jinja templates. For this to work, you need to define **kwargs in your function header.

execute(context)[source]

If the condition evaluates to True execute the superclass execute method, otherwise skip the task.

Parameters:

context (Dict) – Context dict provided by airflow

poke(context)[source]

If the condition evaluates to True execute the superclass poke method, otherwise skip the task.

Parameters:

context (Dict) – Context dict provided by airflow

Return type:

bool

Returns:

The result of the superclass poke method

template_fields = ('condition_callable', 'condition_args', 'condition_kwargs')
class sai_airflow_plugins.operators.conditional_operators.ConditionalBashOperator(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, BashOperator

Conditional bash operator.

See also

ConditionalSkipMixin and BashOperator

template_ext: Sequence[str] = ('.sh', '.bash')
template_fields: Sequence[str] = ('bash_command', 'env', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#ede4ff'
class sai_airflow_plugins.operators.conditional_operators.ConditionalFabricOperator(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, FabricOperator

Conditional Fabric operator.

See also

ConditionalSkipMixin and FabricOperator

template_ext: Sequence[str] = ('.sh',)
template_fields: Sequence[str] = ('ssh_conn_id', 'command', 'remote_host', 'environment', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#feffe5'
class sai_airflow_plugins.operators.conditional_operators.ConditionalPythonOperator(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, PythonOperator

Conditional python operator.

See also

ConditionalSkipMixin and PythonOperator

template_fields: Sequence[str] = ('templates_dict', 'op_args', 'op_kwargs', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#ffebff'
class sai_airflow_plugins.operators.conditional_operators.ConditionalTriggerDagRunOperator(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, TriggerDagRunOperator

Conditional trigger DAG run operator.

See also

ConditionalSkipMixin and TriggerDagRunOperator

template_fields: Sequence[str] = ('trigger_dag_id', 'trigger_run_id', 'execution_date', 'conf', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#efeaff'

sai_airflow_plugins.sensors

class sai_airflow_plugins.sensors.fabric_sensor.FabricSensor(*args, **kwargs)[source]

Bases: BaseSensorOperator, FabricOperator

Executes a command on a remote host using the [Fabric](https://www.fabfile.org) library and returns True if and only if the exit code is 0. Like FabricOperator it uses a standard SSHHook for the connection configuration.

The parameters for this sensor are the combined parameters of FabricOperator and BaseSensorOperator.

poke(context)[source]

Executes self.command over the configured SSH connection and checks its exit code.

Parameters:

context (Dict) – Context dict provided by airflow

Return type:

bool

Returns:

True if the command’s exit code was 0, else False.

template_ext: Sequence[str] = ('.sh',)
template_fields: Sequence[str] = ('ssh_conn_id', 'command', 'remote_host', 'environment')
class sai_airflow_plugins.sensors.conditional_sensors.ConditionalBashSensor(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, BashSensor

Conditional bash sensor.

See also

ConditionalSkipMixin and BashSensor

template_ext: Sequence[str] = ()
template_fields: Sequence[str] = ('bash_command', 'env', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#ede4ff'
class sai_airflow_plugins.sensors.conditional_sensors.ConditionalFabricSensor(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, FabricSensor

Conditional Fabric sensor.

See also

ConditionalSkipMixin and FabricSensor

template_ext: Sequence[str] = ('.sh',)
template_fields: Sequence[str] = ('ssh_conn_id', 'command', 'remote_host', 'environment', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#e6f2eb'
class sai_airflow_plugins.sensors.conditional_sensors.ConditionalPythonSensor(condition_callable=False, condition_args=None, condition_kwargs=None, condition_provide_context=False, *args, **kwargs)[source]

Bases: ConditionalSkipMixin, PythonSensor

Conditional python sensor.

See also

ConditionalSkipMixin and bash_sensorPythonSensor

template_fields: Sequence[str] = ('templates_dict', 'op_args', 'op_kwargs', 'condition_callable', 'condition_args', 'condition_kwargs')
ui_color: str = '#ffebff'