OK, but what are you going to do about the interactive REPL?
On Tue, Nov 17, 2015 at 7:40 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 05:00, Guido van Rossum wrote:
>>>> If you free the memory used for the source buffer before starting code
>> generation you should be good.
>>> Thank you. The buffer is freed just after the end of generating AST.
>>>> On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>>>> I'm working on rewriting Python tokenizer (in particular the part that
>>> reads
>>> and decodes Python source file). The code is complicated. For now there
>>> are
>>> such cases:
>>>>>> * Reading from the string in memory.
>>> * Interactive reading from the file.
>>> * Reading from the file:
>>> - Raw reading ignoring encoding in parser generator.
>>> - Raw reading UTF-8 encoded file.
>>> - Reading and recoding to UTF-8.
>>>>>> The file is read by the line. It makes hard to check correctness of the
>>> first line if the encoding is specified in the second line. And it makes
>>> very hard problems with null bytes and with desynchronizing buffered C
>>> and
>>> Python files. All this problems can be easily solved if read all Python
>>> source file in memory and then parse it as string. This would allow to
>>> drop
>>> a large complex and buggy part of code.
>>>>>> Are there disadvantages in this solution? As for memory consumption, the
>>> source text itself will consume only small part of the memory consumed by
>>> AST tree and other structures. As for performance, reading and decoding
>>> all
>>> file can be faster then by the line.
>>>>>> [1] http://bugs.python.org/issue25643>>>>>> _______________________________________________
>>> Python-Dev mailing list
>>>Python-Dev at python.org>>>https://mail.python.org/mailman/listinfo/python-dev>>> Unsubscribe:
>>>https://mail.python.org/mailman/options/python-dev/guido%40python.org>>>>>>>>>>> _______________________________________________
> Python-Dev mailing list
>Python-Dev at python.org>https://mail.python.org/mailman/listinfo/python-dev> Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/guido%40python.org
--
--Guido van Rossum (python.org/~guido)
