The ansible-core
code runs on both Python 2 and Python 3 because we want Ansible to be able to manage a widevariety of machines. Contributors to ansible-core and to Ansible Collections should be aware of the tips in this document so that they can write code that will run on the same versions of Python as the rest of Ansible.
To ensure that your code runs on Python 3 as well as on Python 2, learn the tips and tricks and idiomsdescribed here. Most of these considerations apply to all three types of Ansible code:
controller-side code - code that runs on the machine where you invoke /usr/bin/ansible
modules - the code which Ansible transmits to and invokes on the managed machine.
shared
module_utils
code - the common code that’s used by modules to perform tasks and sometimes used by controller-side code as well
Ansible Runner is a tool and python library that helps when interfacing with Ansible directly or as part of another system whether that be through a container image interface, as a standalone tool, or as a Python module that can be. This project wraps the ansiblerunner interface inside a REST API enabling ansible playbooks to be executed and queried from other platforms. The incentive for this is two-fold; provide Ansible integration to non-python projects.
Ansible Runner is a tool and python library that helps when interfacing with Ansible directly or as part of another system whether that be through a container image interface, as a standalone tool, or as a Python module that can be imported. The goal is to provide a stable and consistent interface abstraction to Ansible. Ansible Runner is a tool and python library that helps when interfacing with Ansible directly or as part of another system whether that be through a container image interface, as a standalone tool, or as a Python module that can be.
However, the three types of code do not use the same string strategy. If you’re developing a module or some module_utils
code, be sure to read the section on string strategy carefully.
On the controller we support Python 3.5 or greater and Python 2.7 or greater. Module-side, wesupport Python 3.5 or greater and Python 2.6 or greater.
Python 3.5 was chosen as a minimum because it is the earliest Python 3 version adopted as thedefault Python by a Long Term Support (LTS) Linux distribution (in this case, Ubuntu-16.04).Previous LTS Linux distributions shipped with a Python 2 version which users can rely upon insteadof the Python 3 version.
For Python 2, the default is for modules to run on at least Python 2.6. This allowsusers with older distributions that are stuck on Python 2.6 to manage theirmachines. Modules are allowed to drop support for Python 2.6 when one oftheir dependent libraries requires a higher version of Python. This is not aninvitation to add unnecessary dependent libraries in order to force yourmodule to be usable only with a newer version of Python; instead it is anacknowledgment that some libraries (for instance, boto3 and docker-py) willonly function with a newer version of Python.
Note
Python 2.4 Module-side Support:
Support for Python 2.4 and Python 2.5 was dropped in Ansible-2.4. RHEL-5(and its rebuilds like CentOS-5) were supported until April of 2017.Ansible-2.3 was released in April of 2017 and was the last Ansible releaseto support Python 2.4 on the module-side.
The best place to start learning about writing code that supports both Python 2 and Python 3is Lennart Regebro’s book: Porting to Python 3.The book describes several strategies for porting to Python 3. The one we’reusing is to support Python 2 and Python 3 from a single code base
Python 2 and Python 3 handle strings differently, so when you write code that supports Python 3you must decide what string model to use. Strings can be an array of bytes (like in C) orthey can be an array of text. Text is what we think of as letters, digits,numbers, other printable symbols, and a small number of unprintable “symbols”(control codes).
In Python 2, the two types for these (str
for bytes andunicode
for text) are often used interchangeably. When dealing onlywith ASCII characters, the strings can be combined, compared, and convertedfrom one type to another automatically. When non-ASCII characters areintroduced, Python 2 starts throwing exceptions due to not knowing what encodingthe non-ASCII characters should be in.
Python 3 changes this behavior by making the separation between bytes (bytes
)and text (str
) more strict. Python 3 will throw an exception whentrying to combine and compare the two types. The programmer has to explicitlyconvert from one type to the other to mix values from each.
In Python 3 it’s immediately apparent to the programmer when code ismixing the byte and text types inappropriately, whereas in Python 2, code that mixes those typesmay work until a user causes an exception by entering non-ASCII input.Python 3 forces programmers to proactively define a strategy forworking with strings in their program so that they don’t mix text and byte strings unintentionally.
Ansible uses different strategies for working with strings in controller-side code, in:ref: modules <module_string_strategy>, and in module_utils code.
In controller-side code we use a strategy known as the Unicode Sandwich (namedafter Python 2’s unicode
text type). For Unicode Sandwich we know thatat the border of our code and the outside world (for example, file and network IO,environment variables, and some library calls) we are going to receive bytes.We need to transform these bytes into text and use that throughout theinternal portions of our code. When we have to send those strings back out tothe outside world we first convert the text back into bytes.To visualize this, imagine a ‘sandwich’ consisting of a top and bottom layerof bytes, a layer of conversion between, and all text type in the center.
This is a partial list of places where we have to convert to and from byteswhen using the Unicode Sandwich string strategy. It’s not exhaustive butit gives you an idea of where to watch for problems.
In Python 2, reading from files yields bytes. In Python 3, it can yield text.To make code that’s portable to both we don’t make use of Python 3’s abilityto yield text but instead do the conversion explicitly ourselves. For example:
Note
Much of Ansible assumes that all encoded text is UTF-8. At somepoint, if there is demand for other encodings we may change that, but fornow it is safe to assume that bytes are UTF-8.
Writing to files is the opposite process:
Note that we don’t have to catch UnicodeError
here because we’retransforming to UTF-8 and all text strings in Python can be transformed backto UTF-8.
Dealing with filenames often involves dropping back to bytes because on UNIX-likesystems filenames are bytes. On Python 2, if we pass a text string to thesefunctions, the text string will be converted to a byte string inside of thefunction and a traceback will occur if non-ASCII characters are present. InPython 3, a traceback will only occur if the text string can’t be decoded inthe current locale, but it’s still good to be explicit and have code whichworks on both versions:
When you are only manipulating a filename as a string without talking to thefilesystem (or a C library which talks to the filesystem) you can often getaway without converting to bytes:
On the other hand, if the code needs to manipulate the filename and also talkto the filesystem, it can be more convenient to transform to bytes right awayand manipulate in bytes.
Warning
Make sure all variables passed to a function are the same type.If you’re working with something like os.path.join()
which takesmultiple strings and uses them in combination, you need to make sure thatall the types are the same (either all bytes or all text). Mixingbytes and text will cause tracebacks.
The interface of Ibis Paint X download free is simple and intuitive. It has a set of three tools for choosing color, brush size, brush type. Includes several useful layers, such as a layer for drawing, erasing, one for adding text. The toolbar at the bottom has the options for saving, sharing, undoing, redoing your painting. Download now ibis Paint X on your Mac. Don't wait any longer and download ibis Paint X on your Mac for free. Install the Android Bluestacks Emulator and enjoy all the Google Play games and applications on your macOS system. Download Bluestacks for MAC. Home Apps Art & Design ibis Paint X MAC. Ibis paint x download for computer.
Interacting with other programs goes through the operating system andC libraries and operates on things that the UNIX kernel defines. Theseinterfaces are all byte-oriented so the Python interface is byte oriented aswell. On both Python 2 and Python 3, byte strings should be given to Python’ssubprocess library and byte strings should be expected back from it.
One of the main places in Ansible’s controller code that we interact withother programs is the connection plugins’ exec_command
methods. Thesemethods transform any text strings they receive in the command (and argumentsto the command) to execute into bytes and return stdout and stderr as byte stringsHigher level functions (like action plugins’ _low_level_execute_command
)transform the output into text strings.
Ansible Runner Python Download
In modules we use a strategy known as Native Strings. This makes thingseasier on the community members who maintain so many of Ansible’smodules, by not breaking backwards compatibility bymandating that all strings inside of modules are text and converting betweentext and bytes at the borders.
Native strings refer to the type that Python uses when you specify a barestring literal:
In Python 2, these are byte strings. In Python 3 these are text strings. Modules should becoded to expect bytes on Python 2 and text on Python 3.
In module_utils
code we use a hybrid string strategy. Although Ansible’smodule_utils
code is largely like module code, some pieces of it areused by the controller as well. So it needs to be compatible with modulesand with the controller’s assumptions, particularly the string strategy.The module_utils code attempts to accept native strings as inputto its functions and emit native strings as their output.
In module_utils
code:
Functions must accept string parameters as either text strings or byte strings.
Functions may return either the same type of string as they were given or the native string type for the Python version they are run on.
Functions that return strings must document whether they return strings of the same type as they were given or native strings.
Module-utils functions are therefore often very defensive in nature.They convert their string parameters into text (using ansible.module_utils.common.text.converters.to_text
)at the beginning of the function, do their work, and then convertthe return values into the native string type (using ansible.module_utils.common.text.converters.to_native
)or back to the string type that their parameters received.
Use the following boilerplate code at the top of all python filesto make certain constructs act the same way on Python 2 and Python 3:
__metaclass__=type
makes all classes defined in the file into new-styleclasses without explicitly inheriting from object
.
The __future__
imports do the following:
Makes imports look in sys.path
for the modules beingimported, skipping the directory in which the module doing the importinglives. If the code wants to use the directory in which the module doingthe importing, there’s a new dot notation to do so.
Makes division of integers always return a float. If you need tofind the quotient use x//y
instead of x/y
.
Changes print
from a keyword into a function.
See also
Since mixing text and bytes types leads to tracebacks we want to be clearabout what variables hold text and what variables hold bytes. We do this byprefixing any variable holding bytes with b_
. For instance:
We do not prefix the text strings instead because we only operateon byte strings at the borders, so there are fewer variables that need bytesthan text.
The third-party Python six library existsto help projects create code that runs on both Python 2 and Python 3. Ansibleincludes a version of the library in module_utils so that other modules can use itwithout requiring that it is installed on the remote system. To make use ofit, import it like this:
Note
Ansible can also use a system copy of six
Ansible will use a system copy of six if the system copy is a laterversion than the one Ansible bundles.
In order for code to function on Python 2.6+ and Python 3, use thenew exception-catching syntax which uses the as
keyword:
Do not use the following syntax as it will fail on every version of Python 3:
In Python 2.x, octal literals could be specified as 0755
. In Python 3,octals must be specified as 0o755
.
Starting in Python 2.6, strings gained a method called format()
to putstrings together. However, one commonly used feature of format()
wasn’tadded until Python 2.7, so you need to remember not to use it in Ansible code:
Both of the format strings above map positional arguments of the format()
method into the string. However, the first version doesn’t work inPython 2.6. Always remember to put numbers into the placeholders so the codeis compatible with Python 2.6.
See also
Python documentation on format strings
In Python 3.x, byte strings do not have a format()
method. However, itdoes have support for the older, percent-formatting.
Note
Percent formatting added in Python 3.5
Ansible Run Python
Percent formatting of byte strings was added back into Python 3 in 3.5.This isn’t a problem for us because Python 3.5 is our minimum version.However, if you happen to be testing Ansible code with Python 3.4 orearlier, you will find that the byte string formatting here won’t work.Upgrade to Python 3.5 to test.
See also
Python documentation on percent formatting
Ansible modules are slightly harder to code to support Python 3 than normal code from other projects. A lot of mocking has to go into unit testing an Ansible module, so it’s harder to test that your changes have fixed everything or to to make sure that later commits haven’t regressed the Python 3 support. Review our testing pages for more information.
SUMMARY
At this moment, Ansible would fail to run on most remote hosts that do only have python3 installed by default, like Fedora 27 or newer, unless the user exclicitely adds the ansible_python_interpreter=python3
inside the inventory.
This is a real issue because it assumes that the user already knows which is the default ansible interpreter on that host, which may not be something he knows or even controls. Think about dynamic inventories or playbooks that would upgrade a host from a version that had python2 to one that that has python3.
ISSUE TYPE
- Feature Idea
COMPONENT NAME
core
ADDITIONAL INFORMATION
This issue was aggravated by the recommandation made https://www.python.org/dev/peps/pep-0394/ which indirectly made newer distributions which ship with python3 by default to not create a symlink from python
to python3
, even when there was no python2 installed.
As it seems highly unlikely to be able to update the PEP with new recomandations, we should look into findina solution for ansible, one that does not cripple the UX and avoids the need to tune ansible_python_interpreter variable for each host. This should happen under the hood by detecting a missing interpreter and falling back to python3 one (caching may be desired too).
Related links
- https://github.com/python/peps/pull/630 - Last merged change to PEP-0394 (April 2018)
- https://github.com/python/peps/pull/785 - Request to update PEP-0394
- https://bugzilla.redhat.com/show_bug.cgi?id=1630882 - Request to add python symlink to Fedora
- https://wiki.ubuntu.com/Python/3 - Ubuntu not creating the symlink because of PEP-0394