Drop Python 2

[davidism]

Once this is merged into master, I'll probably release a 3.0.0a1 so people can start testing early.

• Remove _compat module and other compat code.
• Run pyupgrade, use f-strings everywhere, replace x and y or z with y if x else z.
• Remove deprecated code from Jinja 2.x.
• Stop building universal wheel.
• Bump bytecode cache version.
• I18N doesn't look for ugettext.
• Remove old docs.

Still have some things left that I'll either hit later or get help with at the PyCascades 2020 sprint:

• Inline asyncsupport instead of patching where possible. Keep in mind performance reasons for patching.
• Refactor compiler to extract some helper functions so things like "async for" if self.is_async else "for", Markup if ctx.volatile else str, etc. aren't written by hand all over the place.
• Use super instead of BaseClass.method where appropriate.
• Apply pyupgrade and code style fixes to docs.
• Apply pyupgrade and code style fixes to generated code.
• Add tests for using Babel for translations.

[mitsuhiko]

Some general notes: You already know my stance that I'm strongly against dropping 2.x support this early for a popular library.

Other things:

• async filter patching is optional because not everybody needs async. We even merged a PR to defer the initialization to when it's actually needed for the first time: [Request] Could importing asyncio optional? #765
• use f-strings everywhere: This does not just drop 2.x support but also 3.5 which is still an intermediate step for some users upgrading from 2.x
• I18N doesn't look for ugettext: strong -1 on this since ugettext is an API contract. There are people who still have wrappers that use this name and there is no benefit to removing it since we already fall back to gettext. This is just going to cause user frustration for absolutely no benefit.
• Use super instead of BaseClass.method, fix inheritance in these cases: inheritance is not broken in the first place. If you want to use it for consistent code style then that's one thing, but don't call the existing behavior broken because it's not. Except for mixins which carefully pick between using super vs baseclass access nothing must even be contained with star inheritance. super() will not fix anything here either but is more likely to break something in fact.

[ThiefMaster]

Does 3.5 have any widespread usage? All major distributions seem to have 3.6/3.7 nowadays. (iirc. dropping 3.5 to be able to use fstrings was actually something I suggested)

What's the reason against having a major release switching to 3.6+? I guess distributions that come with an older Python version wouldn't update to 3.0 anyway, while those that do come with 3.6+, can safely update to 3.0.

[mitsuhiko]

@ThiefMaster might be worth checking how much of this is still true but 3.5 used to be a good in-between target for a long time because it did not reject invalid regex escapes yet.

[davidism]

Some general notes: You already know my stance that I'm strongly against dropping 2.x support this early for a popular library.

Yes, not planning to release this until April at least, but I want to get it into master early so I can make alpha releases for people to test.

async filter patching is optional because not everybody needs async. We even merged a PR to defer the initialization to when it's actually needed for the first time: #765

You're right, I worded this wrong. I more meant "investigate what can be done now that we don't need to defer it for compatibility reasons, but I don't mean to force everything to use async, there will still be the patch_all function in some form.

use f-strings everywhere: This does not just drop 2.x support but also 3.5 which is still an intermediate step for some users upgrading from 2.x

We're already dropping Python 3.5 support, as outlined in the blog post. Users should still be able to use 2.11.x if they need to use Python 3.5 as an intermediary upgrade.

I18N doesn't look for ugettext: strong -1 on this since ugettext is an API contract. There are people who still have wrappers that use this name and there is no benefit to removing it since we already fall back to gettext. This is just going to cause user frustration for absolutely no benefit.

OK, I'll issue a deprecation warning for this.

Use super instead of BaseClass.method, fix inheritance in these cases: inheritance is not broken in the first place. If you want to use it for consistent code style then that's one thing, but don't call the existing behavior broken because it's not. Except for mixins which carefully pick between using super vs baseclass access nothing must even be contained with star inheritance. super() will not fix anything here either but is more likely to break something in fact.

Sorry, I worded this really poorly. I didn't mean to imply there was something wrong or broken with the current code. I meant "consistent code style," using super() consistently where appropriate.

[mitsuhiko]

OK, I'll issue a deprecation warning for this.

Is there a reason to drop support at all for this? It does not seem to add either development or performance overhead.

[davidism]

The intention was to bring it in line with the Python 3 API. However, I can see how this would be an issue if whatever I18N library was still providing 2/3 compatibility. I'll roll this back with a comment about why it remains.

[davidism]

Just noticed that we don't have any tests that use Babel for translations. I was confused for a second when I realized that all our tests were still passing even though Babel would presumably expect us to use ugettext.

[mitsuhiko]

@davidism the last comment is a pretty good example of why we should not touch code for the purpose of just making a self justifying refactor ;)

There will always stuff that breaks and nothing is more frustrating for a user to see that their regression came from a commit like this.

[kevin-brown]

Unpopular opinion: let's rip the band-aid off and get this done with.

You already know my stance that I'm strongly against dropping 2.x support this early for a popular library.

Yes, not planning to release this until April at least, but I want to get it into master early so I can make alpha releases for people to test.

I definitely agree with getting this into the master branch early to ensure that it gets proper testing. We know this will be a breaking change, especially considering we are planning on bumping the major version, so let's get as many eyes as possible on it.

Yes, people are still using Python 2.7. People are also using Python 2.6 (I used to work for a company which did) but you don't see people arguing that we should continue supporting it. RHEL 6 is still supported and that ships with Python 2.6 out of the box.

[wgwz]

I work at a fairly large organization that uses python3.5. We use primarily el6 and el7. I'm not exactly sure of the default version of python3 that ships with those. But we have production systems on both using 3.5. Despite the fact that this is anecdotal I am sure my team is not unique. Recently a tox dependency called zipp broke our build process by dropping support for 3.5. Suffice to say I was not too happy about it. But that was a relatively small dependency. If something like that were to happen with jinja, I would be even more upset.

[kevin-brown]

For what it's worth, I'm in favor of dropping Python 2 because it's legacy Python and officially EOL. I'm indifferent on Python 3.5 and other versions in the 3 line that aren't EOL yet.

I work at a fairly large organization that uses python3.5.

Python 3.5 is approaching EOL and is only receiving security patches right now.

We use primarily el6 and el7. I'm not exactly sure of the default version of python3 that ships with those.

• RHEL 6 ships with Python 2.6
• RHEL 7 ships with Python 2.7
• RHEL 8 ships with Python 3.6 when you opt to install it

Recently a tox dependency called zipp broke our build process by dropping support for 3.5.

In theory, setting the python_requires to not allow Python 3.5 should prevent the package itself from being installed by pip. zipp appears to have done this so I'm not sure what went wrong in your case.

[wgwz]

In theory, setting the python_requires to not allow Python 3.5 should prevent the package itself from being installed by pip. zipp appears to have done this so I'm not sure what went wrong in your case.

The specifics of my situation don't really matter for this discussion. Suffice to say if you have tox as a build dependency, you have not pinned zipp and you are using 3.5, you will not be able to install tox. What matters is that the error that got raised in my situation was about f-strings.

If this MR indeed breaks compatibility with 3.5 just because of use of f-strings, that seems like a mistake to me. If there is a more substantial reason to break with 3.5 then I would be willing to change my opinion.

[davidism]

Dropping support doesn't mean you can't continue to download existing compatible versions. 2.11 still exists, you'll still be able to install it on Python 3.5, and can use that as a pin or as a step during an upgrade.

Regardless of our timeline for Python 3.5 (or our use of f-strings), it goes EOL very soon. We dropped 3.4 in a similar way. You're going to have to deal with dropped support one way or another, from Jinja and/or the language, so it's better to pin now to 2.11 (which will still receive security/bug fixes during that time), and start addressing what you plan to do in general if dropped support is an issue.

You're taking on the risk of a language version that will never get updates (2.7) or will not in a few months (3.5). The risk of Jinja not getting updates either is comparatively minimal, especially given the fact that it's stable and I've said I will accept fixes to 2.11.x and merge them forward for a while. Given how rarely such issues or fixes are actually submitted by the community, I don't expect much churn.

[mitsuhiko]

Dropping support doesn't mean you can't continue to download existing compatible versions. 2.11 still exists, you'll still be able to install it on Python 3.5, and can use that as a pin or as a step during an upgrade.

I want to re-iterate that this is a major departure to how Jinja2 was maintained in the past. I kept 2.4 and 2.5 support alive long after they stopped being supported because CentOS users kept getting supported Python versions.

The argument that we have to drop support because Python itself stops being updated is one that I did not subscribe to back then and I still don't do it. It does not make maintenance of Jinja any easier (if anything it makes it worse because now we need to maintain two branches) and it also does not add anything to the user.

2.x will stay around for years to come. Why do we commit ourselves to having to maintain two branches for years? RedHat and Centos will carry Python 2 until 2024 and I expect large 2.x deployments to stay around for at least that period of time.

I understand I'm not very active on this project any more but this departure from the status quo I am not happy about and it's inconsistent with how I used to develop this library.

[mitsuhiko]

In theory, setting the python_requires to not allow Python 3.5 should prevent the package itself from being installed by pip. zipp appears to have done this so I'm not sure what went wrong in your case.

That's not very useful. It just prevents installing the package. It does not cause pip to auto select a compatible version.

[ThiefMaster]

AFAIK it should (for any somewhat recent pip version), unless the package was built with distribute instead of setuptools (or possibly uploaded with setup.py upload instead of twine upload).