PEP 798: Add Discussion of Alternative GenExp Semantics

peps/pep-0798.rst

@@ -119,14 +119,19 @@ existing syntax ``[x for it in its for x in it]`` is one that students often
 get wrong, the natural impulse for many students being to reverse the order of
 the ``for`` clauses.
 
+Additionally, the comment section of a `Reddit post
+<https://old.reddit.com/r/Python/comments/1m607oi/pep_798_unpacking_in_comprehensions/>`__
+following the publication of this PEP shows substantial support for the
+proposal and further suggests that the syntax proposed here is legible,
+intuitive, and useful.
 
 Specification
 =============
 
 Syntax
 ------
 
-The necessary grammatical changes are allowing the expression in list/set
+The grammar should be changed to allow the expression in list/set
 comprehensions and generator expressions to be preceded by a ``*``, and
 allowing an alternative form of dictionary comprehension in which a
 double-starred expression can be used in place of a ``key: value`` pair.
@@ -204,29 +209,36 @@ respectively::
     for x in dicts:
         new_dict.update(expr)
 
+.. _pep798-genexpsemantics:
 
 Semantics: Generator Expressions
 --------------------------------
 
-A generator expression ``(*expr for x in it)`` forms a generator producing
-values from the concatenation of the iterables given by the expressions.
-Specifically, the behavior is defined to be equivalent to the following
-generator::
+Generator expressions using the unpacking syntax should form new generators
+producing values from the concatenation of the iterables given by the
+expressions.  Specifically, the behavior is defined to be equivalent to the
+following (though without defining or referencing the looping variable
+``i``)::
 
+    # equivalent to generator = (*expr for x in it)
     def generator():
         for x in it:
-            yield from expr
-
-Since ``yield from`` is not allowed inside of async generators (see the section
-of :pep:`525` on Asynchronous ``yield from``), the equivalent for ``(*expr
-async for x in ait())`` is more like the following (though of course this new
-form should not define or reference the looping variable ``i``)::
+            for i in expr:
+                yield i
 
+    # equivalent to generator = (*expr for x in ait())
     async def generator():
         async for x in ait():
             for i in expr:
                 yield i
 
+
+The specifics of these semantics should be revisited in the future,
+particularly if async generators receive support for ``yield from`` (in which
+case both forms may wish to be changed to make use of ``yield from`` instead of
+an explicit loop).  See :ref:`pep798-alternativegenexpsemantics` for more
+discussion.
+
 Interaction with Assignment Expressions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -244,7 +256,8 @@ form, ``y`` will be bound in the containing scope instead of locally::
 
     def generator():
         for i in (0, 2, 4):
-            yield from (y := [i, i+1])
+            for j in (y := [i, i+1]):
+                yield j
 
 In this example, the subexpression ``(y := [i, i+1])`` is evaluated exactly
 three times before the generator is exhausted: just after assigning ``i`` in
@@ -362,9 +375,9 @@ cases:
 Reference Implementation
 ========================
 
-A `reference implementation <https://github.com/adqm/cpython/tree/comprehension_unpacking>`_
-is available, which implements this functionality, including draft documentation and
-additional test cases.
+The `reference implementation <https://github.com/adqm/cpython/tree/comprehension_unpacking>`_
+implements this functionality, including draft documentation and additional
+test cases.
 
 Backwards Compatibility
 =======================
@@ -572,7 +585,7 @@ expressions that involve unpacking::
             yield expr
     g = generator()
 
-    # equivalent to g = (*expr for x in it)
+    # roughly equivalent to g = (*expr for x in it)
     def generator():
         for x in it:
             yield from expr
@@ -642,7 +655,7 @@ resulting generator, but several alternatives were suggested in our discussion
   other aspects of this proposal are accepted.
 
 The reason to prefer this proposal over these alternatives is the preservation
-of existent conventions for punctuation around generator expressions.
+of existing conventions for punctuation around generator expressions.
 Currently, the general rule is that generator expressions must be wrapped in
 parentheses except when provided as the sole argument to a function, and this
 proposal suggests maintaining that rule even as we allow more kinds of
@@ -700,6 +713,65 @@ PEP.  As such, these forms should continue to raise a ``SyntaxError``, but with
 a new error message as described above, though it should not be ruled out as a
 consideration for future proposals.
 
+.. _pep798-alternativegenexpsemantics:
+
+Alternative Generator Expression Semantics
+------------------------------------------
+
+Another point of discussion centered around the semantics of unpacking in
+generator expressions, particularly the relationship between the semantics of
+synchronous and asynchronous generator expressions given that the latter do not
+support ``yield from`` (see the section of :pep:`525` on Asynchronous ``yield
+from``).
+
+Several reasonable options were considered, none of which was a clear winner in
+a `poll in the Discourse thread
+<https://discuss.python.org/t/pep-798-unpacking-in-comprehensions/99435/33>`__:
+
+1. Using ``yield from`` for unpacking in synchronous generator expressions but
+   not in asynchronous generator expressions (as proposed in the original draft
+   of this PEP).
+
+   This strategy would have allowed unpacking in generator expressions to
+   closely mimic a popular way of writing generators that perform this
+   operation (using ``yield from``), but it would also have created an
+   asymmetry between synchronous and asynchronous versions, and also between
+   this new syntax and ``itertools.chain`` and the double-loop version.
+
+2. Using ``yield from`` for unpacking in synchronous generator expressions and
+   mimicking the behavior of ``yield from`` for unpacking in async generator
+   expressions.
+
+   This strategy would also make unpacking in synchronous and asynchronous
+   generators behave similarly, but it would also be more complex, enough so
+   that the cost may not be worth the benefit, particularly in the absence of a
+   compelling practical use case for delegating to subgenerators during
+   unpacking.
+
+3. Disallowing unpacking in asynchronous generator expressions until they
+   support ``yield from``.
+
+   This strategy could possibly reduce friction if asynchronous generator
+   expressions do gain support for ``yield from`` in the future, but in the
+   meantime, it would result in an even bigger discrepancy between synchronous
+   and asynchronous generator expressions than option 1.
+
+4. Disallowing unpacking in all generator expressions.
+
+   This would retain symmetry between the two cases, but with the downside of
+   losing a very expressive form.
+
+Each of these options (including the one presented in this PEP) has its
+benefits and drawbacks, with no option being clearly superior on all fronts;
+but the semantics proposed in :ref:`pep798-genexpsemantics` represent a
+reasonable compromise.
+
+As suggested above, this decision should be revisited in the event that
+asynchronous generators receive support for ``yield from`` in the future,
+in which case the ability to delegate to subgenerators during unpacking
+could be added without significant cost.
+
+
 Concerns and Disadvantages
 ==========================
 
@@ -722,7 +794,7 @@ were raised as well. This section aims to summarize those concerns.
   Complex uses of unpacking in comprehensions could obscure logic that would be
   clearer in an explicit loop.  While this is already a concern with
   comprehensions more generally, the addition of ``*`` and ``**`` may make
-  particularly-complex uses even more difficult to read and understand at a
+  particularly complex uses even more difficult to read and understand at a
   glance.  For example, while these situations are likely rare, comprehensions
   that use unpacking in multiple ways can make it difficult to know what's
   being unpacked and when: ``f(*(*x for *x, _ in list_of_lists))``.
@@ -768,7 +840,7 @@ Many languages that support comprehensions support double loops:
     (for [xs [[1 2 3] [] [4 5]] x (concat xs xs)] x)
 
 Several other languages (even those without comprehensions) support these
-operations via a built-in function/method to support flattening of nested
+operations via a built-in function or method to support flattening of nested
 structures:
 
 .. code:: python
@@ -778,7 +850,7 @@ structures:
 
 .. code:: javascript
 
-    // Javascript
+    // JavaScript
     [[1,2,3], [], [4,5]].flatMap(xs => [...xs, ...xs])
 
 .. code:: haskell
@@ -801,7 +873,7 @@ in Julia currently leads to a syntax error:
 
 As one counterexample, support for a similar syntax was recently added to `Civet
 <https://civet.dev/>`_.  For example, the following is a valid comprehension in
-Civet, making use of Javascript's ``...`` syntax for unpacking:
+Civet, making use of JavaScript's ``...`` syntax for unpacking:
 
 .. code:: javascript