Mailman 3 March 2014 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

More compact dictionaries with faster iteration
by Raymond Hettinger Jan. 3, 2015

Jan. 3, 2015

The current memory layout for dictionaries is unnecessarily inefficient. It has a sparse table of 24-byte entries containing the hash value, key pointer, and value pointer. Instead, the 24-byte entries should be stored in a dense table referenced by a sparse table of indices. For example, the dictionary: d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'} is currently stored as: entries = [['--', '--', '--'], [-8522787127447073495, 'barry', 'green'], ['--', '--', '--'], ['--', '--', '--'], ['--', '--', '--'], [-9092791511155847987, 'timmy', 'red'], ['--', '--', '--'], [-6480567542315338377, 'guido', 'blue']] Instead, the data should be organized as follows: indices = [None, 1, None, None, None, 0, None, 2] entries = [[-9092791511155847987, 'timmy', 'red'], [-8522787127447073495, 'barry', 'green'], [-6480567542315338377, 'guido', 'blue']] Only the data layout needs to change. The hash table algorithms would stay the same. All of the current optimizations would be kept, including key-sharing dicts and custom lookup functions for string-only dicts. There is no change to the hash functions, the table search order, or collision statistics. The memory savings are significant (from 30% to 95% compression depending on the how full the table is). Small dicts (size 0, 1, or 2) get the most benefit. For a sparse table of size t with n entries, the sizes are: curr_size = 24 * t new_size = 24 * n + sizeof(index) * t In the above timmy/barry/guido example, the current size is 192 bytes (eight 24-byte entries) and the new size is 80 bytes (three 24-byte entries plus eight 1-byte indices). That gives 58% compression. Note, the sizeof(index) can be as small as a single byte for small dicts, two bytes for bigger dicts and up to sizeof(Py_ssize_t) for huge dict. In addition to space savings, the new memory layout makes iteration faster. Currently, keys(), values, and items() loop over the sparse table, skipping-over free slots in the hash table. Now, keys/values/items can loop directly over the dense table, using fewer memory accesses. Another benefit is that resizing is faster and touches fewer pieces of memory. Currently, every hash/key/value entry is moved or copied during a resize. In the new layout, only the indices are updated. For the most part, the hash/key/value entries never move (except for an occasional swap to fill a hole left by a deletion). With the reduced memory footprint, we can also expect better cache utilization. For those wanting to experiment with the design, there is a pure Python proof-of-concept here: http://code.activestate.com/recipes/578375 YMMV: Keep in mind that the above size statics assume a build with 64-bit Py_ssize_t and 64-bit pointers. The space savings percentages are a bit different on other builds. Also, note that in many applications, the size of the data dominates the size of the container (i.e. the weight of a bucket of water is mostly the water, not the bucket). Raymond

27 70

Semi-official read-only Github mirror of the CPython Mercurial repository
by Eli Bendersky Oct. 15, 2014

Oct. 15, 2014

Hi all, https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns. I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome. The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went down from almost ~2GB to ~200MB (so this is the size of a fresh clone right now). Eli P.S. thanks Jesse for the keys to https://github.com/python

4 8

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

19 35

Status of PEP 3145 - Asynchronous I/O for subprocess.popen
by Antoine Pitrou May 30, 2014

May 30, 2014

Hi, On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for subprocess.popen. I answered that asyncio now has subprocess support (including non-blocking I/O on the three standard stream pipes), so it's not obvious anything else is needed. Should we change the PEP's status to Rejected or Superseded? Regards Antoine.

11 35

Summary of Python tracker Issues
by Python tracker May 15, 2014

May 15, 2014

ACTIVITY SUMMARY (2014-01-31 - 2014-02-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4501 (+17) closed 27813 (+65) total 32314 (+82) Open issues with patches: 2045 Issues opened (65) ================== #12915: Add inspect.locate and inspect.resolve http://bugs.python.org/issue12915 reopened by vinay.sajip #16779: Fix compiler warning when building extension modules on 64-bit http://bugs.python.org/issue16779 reopened by skrah #19363: Python 2.7's future_builtins.map is not compatible with Python http://bugs.python.org/issue19363 reopened by Gareth.Rees #20053: venv and ensurepip are affected by default pip config file http://bugs.python.org/issue20053 reopened by ncoghlan #20461: Argument Clinic included return converters hard code use of `` http://bugs.python.org/issue20461 opened by brett.cannon #20462: Python IDLE auto closed when creating a new file or open exist http://bugs.python.org/issue20462 opened by Bishop #20463: sqlite dumpiter dumps invalid script when virtual tables are u http://bugs.python.org/issue20463 opened by Ronny.Pfannschmidt #20464: Update distutils sample config file in Doc/install/index.rst http://bugs.python.org/issue20464 opened by zach.ware #20465: Upgrade SQLite to 3.8.3 with 3.4.0 Windows and OS X installers http://bugs.python.org/issue20465 opened by ned.deily #20466: Example in Doc/extending/embedding.rst fails to compile cleanl http://bugs.python.org/issue20466 opened by zach.ware #20467: Confusing wording about __init__ http://bugs.python.org/issue20467 opened by BreamoreBoy #20468: resource module documentation is incorrect http://bugs.python.org/issue20468 opened by Kurt.Rose #20469: ssl.getpeercert() should include extensions http://bugs.python.org/issue20469 opened by oninoshiko #20475: pystone.py in 3.4 still uses time.clock(), even though it's ma http://bugs.python.org/issue20475 opened by pfalcon #20476: If new email policies are used, default message factory should http://bugs.python.org/issue20476 opened by r.david.murray #20478: Avoid inadvertently special casing Counter in statistics modul http://bugs.python.org/issue20478 opened by ncoghlan #20479: Efficiently support weight/frequency mappings in the statistic http://bugs.python.org/issue20479 opened by ncoghlan #20480: Add ipaddress property to get name of reverse DNS PTR record http://bugs.python.org/issue20480 opened by leonn #20481: Clarify type coercion rules in statistics module http://bugs.python.org/issue20481 opened by ncoghlan #20482: smtplib.SMTP.sendmail: improve exception message http://bugs.python.org/issue20482 opened by yegle #20483: Missing network resource checks in test_urllib2 & test_smtplib http://bugs.python.org/issue20483 opened by ncoghlan #20484: test_pydoc can alter execution environment causing subsequent http://bugs.python.org/issue20484 opened by ned.deily #20485: Enable non-ASCII extension module names http://bugs.python.org/issue20485 opened by Suzumizaki #20486: msilib: can't close opened database http://bugs.python.org/issue20486 opened by gentoo90 #20487: Odd words in unittest.mock document. http://bugs.python.org/issue20487 opened by naoki #20490: Show clear error message on circular import http://bugs.python.org/issue20490 opened by cool-RR #20491: textwrap: Non-breaking space not honored http://bugs.python.org/issue20491 opened by kunkku #20493: asyncio: OverflowError('timeout is too large') http://bugs.python.org/issue20493 opened by haypo #20495: test_read_pty_output() hangs on FreeBSD 7.2 buildbot http://bugs.python.org/issue20495 opened by haypo #20496: function definition tutorial encourages bad practice http://bugs.python.org/issue20496 opened by aisaac #20497: Unclear word in socket document. http://bugs.python.org/issue20497 opened by naoki #20499: Rounding errors with statistics.variance http://bugs.python.org/issue20499 opened by oscarbenjamin #20500: assertion failed when passing an exception object to sys.exit http://bugs.python.org/issue20500 opened by xdegaye #20501: fileinput module will read whole file into memory when using f http://bugs.python.org/issue20501 opened by gromgull #20503: super behavioru and abstract base classes (either implementati http://bugs.python.org/issue20503 opened by Gerrit.Holl #20504: cgi.FieldStorage, multipart, missing Content-Length http://bugs.python.org/issue20504 opened by srittau #20505: Remove resolution from selectors and granularity from asyncio http://bugs.python.org/issue20505 opened by haypo #20506: Command to display all available Import Library http://bugs.python.org/issue20506 opened by arupchakrav #20507: TypeError from str.join has no message http://bugs.python.org/issue20507 opened by Gareth.Rees #20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess http://bugs.python.org/issue20508 opened by Gareth.Rees #20510: Test cases in test_sys don't match the comments http://bugs.python.org/issue20510 opened by Gareth.Rees #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 opened by yselivanov #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 opened by vScherba #20515: Null pointer dereference in tkinter module http://bugs.python.org/issue20515 opened by christian.heimes #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 opened by glangford #20517: Support errors with two filenames for errno exceptions http://bugs.python.org/issue20517 opened by larry #20518: multiple inheritance + C extension = possibly unexpected __bas http://bugs.python.org/issue20518 opened by Martin.Teichmann #20519: Replace uuid ctypes usage with an extension module. http://bugs.python.org/issue20519 opened by gustavo #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 opened by serhiy.storchaka #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 opened by svenberkvens #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 opened by mbyt #20524: format error messages should provide context information http://bugs.python.org/issue20524 opened by r.david.murray #20525: Got compiler warning when compiling readline module http://bugs.python.org/issue20525 opened by vajrasky #20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc) http://bugs.python.org/issue20526 opened by haypo #20529: Unittest displays ResourceWarning warnings when running tests, http://bugs.python.org/issue20529 opened by Bernt.Røskar.Brenna #20530: Change the text signature format (again) to be more robust http://bugs.python.org/issue20530 opened by larry #20531: TypeError in e-mail.parser when non-ASCII is present http://bugs.python.org/issue20531 opened by jason.coombs #20532: Mark all tests which use _testcapi as CPython only http://bugs.python.org/issue20532 opened by serhiy.storchaka #20534: Enum tests fail with pickle protocol 4 http://bugs.python.org/issue20534 opened by ethan.furman #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 opened by Arfrever #20536: statistics._decimal_to_ratio() produces non-integer ratio http://bugs.python.org/issue20536 opened by skrah #20537: logging exc_info parameter should accept exception instances http://bugs.python.org/issue20537 opened by yselivanov #20538: Segfault in UTF-7 incremental decoder http://bugs.python.org/issue20538 opened by serhiy.storchaka #20539: math.factorial may throw OverflowError http://bugs.python.org/issue20539 opened by ncoghlan #20540: Python 3.3/3.4 regression in multiprocessing manager ? http://bugs.python.org/issue20540 opened by Irvin.Probst Most recent 15 issues with no replies (15) ========================================== #20536: statistics._decimal_to_ratio() produces non-integer ratio http://bugs.python.org/issue20536 #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 #20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc) http://bugs.python.org/issue20526 #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 #20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess http://bugs.python.org/issue20508 #20504: cgi.FieldStorage, multipart, missing Content-Length http://bugs.python.org/issue20504 #20503: super behavioru and abstract base classes (either implementati http://bugs.python.org/issue20503 #20500: assertion failed when passing an exception object to sys.exit http://bugs.python.org/issue20500 #20497: Unclear word in socket document. http://bugs.python.org/issue20497 #20496: function definition tutorial encourages bad practice http://bugs.python.org/issue20496 #20486: msilib: can't close opened database http://bugs.python.org/issue20486 Most recent 15 issues waiting for review (15) ============================================= #20540: Python 3.3/3.4 regression in multiprocessing manager ? http://bugs.python.org/issue20540 #20537: logging exc_info parameter should accept exception instances http://bugs.python.org/issue20537 #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 #20534: Enum tests fail with pickle protocol 4 http://bugs.python.org/issue20534 #20532: Mark all tests which use _testcapi as CPython only http://bugs.python.org/issue20532 #20530: Change the text signature format (again) to be more robust http://bugs.python.org/issue20530 #20525: Got compiler warning when compiling readline module http://bugs.python.org/issue20525 #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 #20519: Replace uuid ctypes usage with an extension module. http://bugs.python.org/issue20519 #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 #20515: Null pointer dereference in tkinter module http://bugs.python.org/issue20515 #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 Top 10 most discussed issues (10) ================================= #20480: Add ipaddress property to get name of reverse DNS PTR record http://bugs.python.org/issue20480 21 msgs #20485: Enable non-ASCII extension module names http://bugs.python.org/issue20485 17 msgs #20481: Clarify type coercion rules in statistics module http://bugs.python.org/issue20481 14 msgs #20053: venv and ensurepip are affected by default pip config file http://bugs.python.org/issue20053 13 msgs #20185: Derby #17: Convert 49 sites to Argument Clinic across 13 files http://bugs.python.org/issue20185 12 msgs #20475: pystone.py in 3.4 still uses time.clock(), even though it's ma http://bugs.python.org/issue20475 10 msgs #15216: Support setting the encoding on a text stream after creation http://bugs.python.org/issue15216 9 msgs #20505: Remove resolution from selectors and granularity from asyncio http://bugs.python.org/issue20505 9 msgs #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 9 msgs #16296: Patch to fix building on Win32/64 under VS 2010 http://bugs.python.org/issue16296 8 msgs Issues closed (62) ================== #2226: Small _abcoll Bugs / Oddities http://bugs.python.org/issue2226 closed by ncoghlan #7538: HP-UX 11.11 GCC build fails to build modules http://bugs.python.org/issue7538 closed by Arfrever #7717: Compilation fixes for IRIX http://bugs.python.org/issue7717 closed by berker.peksag #11117: Implementing Async IO http://bugs.python.org/issue11117 closed by yselivanov #12187: subprocess.wait() with a timeout uses polling on POSIX http://bugs.python.org/issue12187 closed by haypo #14515: tempfile.TemporaryDirectory documented as returning object but http://bugs.python.org/issue14515 closed by r.david.murray #14665: faulthandler prints tracebacks in reverse order http://bugs.python.org/issue14665 closed by ncoghlan #14854: faulthandler: fatal error with "SystemError: null argument to http://bugs.python.org/issue14854 closed by haypo #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 closed by yselivanov #17162: Py_LIMITED_API needs a PyType_GenericDealloc http://bugs.python.org/issue17162 closed by loewis #18801: inspect.classify_class_attrs() misclassifies object.__new__() http://bugs.python.org/issue18801 closed by yselivanov #18891: Master patch for content manager addition to email package http://bugs.python.org/issue18891 closed by r.david.murray #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 closed by python-dev #19320: Tkinter tests ran with wantobjects is false http://bugs.python.org/issue19320 closed by serhiy.storchaka #19334: test_asyncio hanging for 1 hour (non-AIX version) http://bugs.python.org/issue19334 closed by haypo #19652: test_asyncio: test_subprocess_send_signal() hangs on buildbot http://bugs.python.org/issue19652 closed by haypo #19761: test_tk fails on OS X with multiple test case failures with bo http://bugs.python.org/issue19761 closed by serhiy.storchaka #19863: Missing function attributes in 2.7 docs. http://bugs.python.org/issue19863 closed by mark.dickinson #19920: TarFile.list() fails on some files http://bugs.python.org/issue19920 closed by serhiy.storchaka #20029: asyncio.SubprocessProtocol is missing http://bugs.python.org/issue20029 closed by haypo #20102: shutil._make_zipfile possible resource leak http://bugs.python.org/issue20102 closed by python-dev #20136: Logging: StreamHandler does not use OS line separator. http://bugs.python.org/issue20136 closed by vinay.sajip #20137: Logging: RotatingFileHandler computes string length instead of http://bugs.python.org/issue20137 closed by vinay.sajip #20162: Test test_hash_distribution fails on RHEL 6.5 / ppc64 http://bugs.python.org/issue20162 closed by haypo #20311: epoll.poll(timeout) and PollSelector.select(timeout) must roun http://bugs.python.org/issue20311 closed by haypo #20319: concurrent.futures.wait() can block forever even if Futures ha http://bugs.python.org/issue20319 closed by bquinlan #20354: tracemalloc causes segfault in "make profile-opt" http://bugs.python.org/issue20354 closed by haypo #20363: BytesWarnings triggerred by test suite http://bugs.python.org/issue20363 closed by serhiy.storchaka #20368: Tkinter: handle the null character http://bugs.python.org/issue20368 closed by serhiy.storchaka #20400: Add create_read_pipe_protocol/create_write_pipe_protocol to as http://bugs.python.org/issue20400 closed by haypo #20404: Delayed exception using non-text encodings with TextIOWrapper http://bugs.python.org/issue20404 closed by python-dev #20412: Enum and IntEnum classes are not defined in the documentation http://bugs.python.org/issue20412 closed by python-dev #20423: io.StringIO newline param has wrong default http://bugs.python.org/issue20423 closed by pitrou #20426: Compiling a regex with re.DEBUG should force a recompile http://bugs.python.org/issue20426 closed by pitrou #20435: Discrepancy between io.StringIO and _pyio.StringIO with univer http://bugs.python.org/issue20435 closed by pitrou #20445: HAVE_BROKEN_NICE detected incorrectly due to configure.ac typo http://bugs.python.org/issue20445 closed by George.Kouryachy #20452: test_timeout_rounding() of test_asyncio fails on "x86 Ubuntu S http://bugs.python.org/issue20452 closed by haypo #20455: test_asyncio hangs on Windows http://bugs.python.org/issue20455 closed by haypo #20456: Argument Clinic rollup patch, 2014/01/31 http://bugs.python.org/issue20456 closed by larry #20460: Wrong markup in c-api/arg.rst http://bugs.python.org/issue20460 closed by zach.ware #20470: "Fatal Python error: Bus error" on the SPARC Solaris 10 buildb http://bugs.python.org/issue20470 closed by haypo #20471: test_signature_on_class() of test_inspect fails on "AMD64 Free http://bugs.python.org/issue20471 closed by yselivanov #20472: test_write_pty() of test_asyncio fails on "x86 Tiger 3.x" buil http://bugs.python.org/issue20472 closed by haypo #20473: inspect.Signature no longer handles builtin classes correctly http://bugs.python.org/issue20473 closed by yselivanov #20474: test_socket failures on OS X due to fixed "expected failures" http://bugs.python.org/issue20474 closed by ned.deily #20477: Add examples of using the new contentmanager api to the email http://bugs.python.org/issue20477 closed by r.david.murray #20488: importlib title and introduction are out of date http://bugs.python.org/issue20488 closed by brett.cannon #20489: help() fails for zlib Compress and Decompress objects http://bugs.python.org/issue20489 closed by serhiy.storchaka #20492: test_tcl is failing on windows 2.7 http://bugs.python.org/issue20492 closed by serhiy.storchaka #20494: Backport of http://bugs.python.org/issue11849 to Python 2.7 (M http://bugs.python.org/issue20494 closed by python-dev #20498: Update StringIO newline tests http://bugs.python.org/issue20498 closed by serhiy.storchaka #20502: Context.create_decimal_from_float() inconsistent precision for http://bugs.python.org/issue20502 closed by mark.dickinson #20509: logging.config.fileConfig() docs could link to the config file http://bugs.python.org/issue20509 closed by python-dev #20512: Python3.3 segfaults when using big5hkscs encoding http://bugs.python.org/issue20512 closed by ned.deily #20514: Errno 10013 - ipython notebook http://bugs.python.org/issue20514 closed by ned.deily #20522: pandas version 0.13.0 does not give warning when summing float http://bugs.python.org/issue20522 closed by mark.dickinson #20527: multiprocessing.Queue deadlocks after “reader” process dea http://bugs.python.org/issue20527 closed by sbt #20528: fileinput module will read whole file into memory when using f http://bugs.python.org/issue20528 closed by r.david.murray #20533: bug in float arithmetic? http://bugs.python.org/issue20533 closed by zach.ware #20541: os.path.exists() gives wrong answer for Windows special files http://bugs.python.org/issue20541 closed by ncoghlan #20542: Assertion failure in test_readline http://bugs.python.org/issue20542 closed by ncoghlan #1028088: Cookies without values are silently ignored (by design?) http://bugs.python.org/issue1028088 closed by berker.peksag

3 2

PEP 466 (round 2): Network security enhancements for Python 2.7
by Nick Coghlan April 19, 2014

April 19, 2014

Several significant changes in this revision: - scope narrowed to just Python 2.7 plus permission for commercial redistributors to use the same strategy in their long term support releases - far more explicit that this is about inviting potential corporate contributors to address the situation for the benefit of the overall Python ecosystem, not offering to fix it for them for free - clarified that third party integration testing services would need to be updated to support testing against multiple Python 2.7 minor releases - explicit sections on why I don't think the status quo is sustainable, why I don't think Python 2.8 would actually solve the problem, and why I think a PyPI based solution not only wouldn't solve the problem, but would be rather difficult to get working in the first place - be completely explicit that I am *not* speaking on behalf of Red Hat at this point and have no authority to make commitments on their behalf. Instead, I'm looking for upstream consensus that 1) this is a genuine problem that needs to be solved; 2) we're open to corporate assistance in solving it; and 3) we have a pretty good idea what help we actually want. If all that happens, *then* I can take up the issue internally to try to get us some help in maintaining the proposed solution (hopefully other folks with corporate influence can do the same, and we may actually get some ongoing assistance with upstream maintenance out of this, rather than having our downstream redistributors continue to take us for granted). Diff: http://hg.python.org/peps/rev/2e82209dda21 Updated web version: http://www.python.org/dev/peps/pep-0466/ Advance warning: while I was able to get this revision turned around pretty quickly, future revisions are likely to take a fair bit longer. It was already a rather busy month before I decided to start this discussion on top of everything else :) Cheers, Nick. PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. Accordingly, this PEP relaxes the normal restrictions by allowing enhancements to be applied in Python 2.7 maintenance releases for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers Proposed backports for these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else disclaim support for the use of older versions in roles that involving connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible changes in these modules to Python 2.7 maintenance releases. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that needs to evolve faster than the standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to review and merge corporate contributions that put this policy into effect. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole. Documentation ------------- All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors applying this policy to any Python release series that is either in security fix only mode, or has been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the redistributor. Integration testing ------------------- Third party integration testing services would likely need to start offering users a choice of multiple Python 2.7.x versions to test against, to ensure that the application is correctly degrading gracefully if it attempts to use newer networking features on maintenance releases that are too old to provide them. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, and currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. Now that we're fully aware of the impact the limitations in Python 2 may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements available in Python 3. While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 as its main Python interpreter for its 14.04 LTS release, Fedora still has a lot of work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and those changes won't affect the core tools regardless. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my currently preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support). This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective as a solution to the original problem (the widespread use of the aging network security infrastructure in Python 2). Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While it initially appears to be an attractive and easier to manage approach, there are actually several significant problems with this idea. Firstly, this PEP encompasses a non-trivial portion of the standard library. It's not just the underlying SSL support, but also the libraries for other network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the SSL infrastructure to provide secure links, and that's just the protocols in the standard library. Even if an API compatible ``ssl2`` module was made available, it would need to be imported and injected into ``sys.modules`` as ``ssl`` before importing any other module that needed it. Secondly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than taking the risk of trying to upgrade it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means going around to each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to *opt out* of the security enhancements, which most of them are unlikely to do. Open Questions ============== * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases? Currently we just upgrade to the appropriate OpenSSL maintenance releases, rather than switching to the latest feature release. In particular, is it possible Windows C extensions may be linking against the Python provided OpenSSL module? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list thread [1]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

11 20

PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3
by Ethan Furman April 12, 2014

April 12, 2014

Okay, I included that last round of comments (from late February). Barring typos, this should be the final version. Final comments? ----------------------------------------------------------------------------- PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan(a)stoneleaf.us> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25 Resolution: Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` formats, and ``FTP`` and ``HTTP`` communications, among many others. Proposed semantics for ``bytes`` and ``bytearray`` formatting ============================================================= %-interpolation --------------- All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, ``%g``, etc.) will be supported, and will work as they do for str, including the padding, justification and other related modifiers. The only difference will be that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code `%x`:: b"%x" % val is equivalent to ("%x" % val).encode("ascii") Examples:: >>> b'%4x' % 10 b' a' >>> b'%#4x' % 10 ' 0xa' >>> b'%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. Examples:: >>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' ``%s`` is included for two reasons: 1) `b` is already a format code for ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x code uses ``%s``; however, it is restricted in what it will accept:: - input type supports ``Py_buffer`` [6]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError`` In particular, ``%s`` will not accept numbers (use a numeric format code for that), nor ``str`` (encode it to ``bytes``). Examples:: >>> b'%s' % b'abc' b'abc' >>> b'%s' % 'some string'.encode('utf8') b'some string' >>> b'%s' % 3.14 Traceback (most recent call last): ... TypeError: b'%s' does not accept numbers, use a numeric code instead >>> b'%s' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%s' does not accept 'str', it must be encoded to `bytes` ``%a`` will call ``ascii()`` on the interpolated value. This is intended as a debugging aid, rather than something that should be used in production. Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn`` representation. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or even a rudimentary serialization format when defining ``__bytes__`` would not be appropriate [8]. .. note:: If a ``str`` is passed into ``%a``, it will be surrounded by quotes. Unsupported codes ----------------- ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported. Proposed variations =================== It was suggested to let ``%s`` accept numbers, but since numbers have their own format codes this idea was discarded. It has been suggested to use ``%b`` for bytes as well as ``%s``. This was rejected as not adding any value either in clarity or simplicity. It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%s``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. It has been proposed to have ``%s`` return the ascii-encoded repr when the value is a ``str`` (b'%s' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed. Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka ``str``) based, and it was dropped. Various new special methods were proposed, such as ``__ascii__``, ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. Objections ========== The objections raised against this PEP were mainly variations on two themes:: - the ``bytes`` and ``bytearray`` types are for pure binary data, with no assumptions about encodings - offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 ``str``/``unicode`` text model As was seen during the discussion, ``bytes`` and ``bytearray`` are also used for mixed binary data and ASCII-compatible segments: file formats such as ``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. ``bytes`` and ``bytearray`` already have several methods which assume an ASCII compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name just a few. %-interpolation, with its very restricted mini-language, will not be any more of a nuisance than the already existing methods. Some have objected to allowing the full range of numeric formatting codes with the claim that decimal alone would be sufficient. However, at least two formats (dbf and pdf) make use of non-decimal numbers. Footnotes ========= .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html .. [4] to use a str object in a bytes interpolation, encode it first .. [5] %c is not an exception as neither of its possible arguments are str .. [6] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ .. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

15 51

libpython added to ABI tracker
by Andrey Ponomarenko April 3, 2014

April 3, 2014

Hi, The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. -- Andrey Ponomarenko, NTC IT ROSA.

6 15

Negative timedelta strings
by Steven D'Aprano April 2, 2014

April 2, 2014

There's a furious discussion going on at the python-list mailing list, about negative timedelta strings: py> str(timedelta(0, -1)) '-1 day, 23:59:59' This is documented. It's even documented as being "somewhat unusual". I found a tracker item for it, back in July 2010: http://bugs.python.org/issue9430 and that refers back to a discussion on #python-dev, but (as far as I know) that's not archived anywhere. I had a look on the python-dev mailing list around that time, and couldn't find anything. Does anyone remember the rationale for this behaviour? Is it open to debate or is it now cast in stone? (Personally, I'm not looking to reopen the debate, and if I did, it would be on python-ideas.) Any links to the earlier discussion would be helpful. Thanks, -- Steve

10 13