• shutil.rmtree() fails when used in Fedora (rpm) "mock" environment

    From c.buhtz@c.buhtz@posteo.jp to comp.lang.python on Thu Oct 24 07:33:04 2024
    From Newsgroup: comp.lang.python

    Hello,
    I am upstream maintainer of "Back In Time" [1] investigating an issue a
    distro maintainer from Fedora reported [2] to me.

    On one hand Fedora seems to use a tool called "mock" to build packages
    in a chroot environment.
    On the other hand the test suite of "Back In Time" does read and write
    to the real file system.
    One test fails because a temporary directory is cleaned up using shutil.rmtree(). Please see the output below.

    I am not familiar with Fedora and "mock". So I am not able to reproduce
    this on my own.
    It seems the Fedora maintainer also has no clue how to solve it or why
    it happens.

    Can you please have a look (especially at the line "assert func is
    os.lstat").
    Maybe you have an idea what is the intention behind this error raised by
    an "assert" statement inside "shutil.rmtree()".

    Thanks in advance,
    Christian Buhtz

    [1] -- <https://github.com/bit-team/backintime>
    [2] -- <https://github.com/bit-team/backintime/issues/1911>

    __________________________ General.test_ctor_defaults __________________________
    self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
    """Default values in constructor."""
    with TemporaryDirectory(prefix='bit.') as temp_name:
    test/test_uniquenessset.py:47:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    /usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
    /usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors) /usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    /usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    stack = []
    onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at
    0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
    # Each stack item has four elements:
    # * func: The first operation to perform: os.lstat, os.close or os.rmdir.
    # Walking a directory starts with an os.lstat() to detect
    symlinks; in
    # this case, func is updated before subsequent operations and passed to
    # onexc() if an error occurs.
    # * dirfd: Open file descriptor, or None if we're processing the top-level
    # directory given to rmtree() and the user didn't supply
    dir_fd.
    # * path: Path of file to operate upon. This is passed to
    onexc() if an
    # error occurs.
    # * orig_entry: os.DirEntry, or None if we're processing the top-level
    # directory given to rmtree(). We used the cached stat() of
    the entry to
    # save a call to os.lstat() when walking subdirectories.
    func, dirfd, path, orig_entry = stack.pop()
    name = path if orig_entry is None else orig_entry.name
    try:
    if func is os.close:
    os.close(dirfd)
    return
    if func is os.rmdir:
    os.rmdir(name, dir_fd=dirfd)
    return

    # Note: To guard against symlink races, we use the standard
    # lstat()/open()/fstat() trick.
    assert func is os.lstat
    E AssertionError
    /usr/lib64/python3.13/shutil.py:663: AssertionError

    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From ram@ram@zedat.fu-berlin.de (Stefan Ram) to comp.lang.python on Thu Oct 24 14:27:32 2024
    From Newsgroup: comp.lang.python

    c.buhtz@posteo.jp wrote or quoted:
    Maybe you have an idea what is the intention behind this error raised by
    an "assert" statement inside "shutil.rmtree()".

    Instead of banking on TemporaryDirectory's auto-cleanup, you
    might wanna take a stab at nuking the directory post-test with
    a more bare-bones approach like os.rmdir() or roll your own
    cleanup function.

    Or you could wrap the TemporaryDirectory usage in a try-except
    block to catch and log any hiccups during cleanup, so the
    test can keep truckin' even if that cleanup goes sideways


    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From MRAB@python@mrabarnett.plus.com to comp.lang.python on Thu Oct 24 15:45:47 2024
    From Newsgroup: comp.lang.python

    On 2024-10-24 08:33, Christian Buhtz via Python-list wrote:
    Hello,
    I am upstream maintainer of "Back In Time" [1] investigating an issue a distro maintainer from Fedora reported [2] to me.

    On one hand Fedora seems to use a tool called "mock" to build packages
    in a chroot environment.
    On the other hand the test suite of "Back In Time" does read and write
    to the real file system.
    One test fails because a temporary directory is cleaned up using shutil.rmtree(). Please see the output below.

    I am not familiar with Fedora and "mock". So I am not able to reproduce
    this on my own.
    It seems the Fedora maintainer also has no clue how to solve it or why
    it happens.

    Can you please have a look (especially at the line "assert func is os.lstat").
    Maybe you have an idea what is the intention behind this error raised by
    an "assert" statement inside "shutil.rmtree()".

    Thanks in advance,
    Christian Buhtz

    [1] -- <https://github.com/bit-team/backintime>
    [2] -- <https://github.com/bit-team/backintime/issues/1911>

    __________________________ General.test_ctor_defaults __________________________
    self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
    """Default values in constructor."""
    with TemporaryDirectory(prefix='bit.') as temp_name:
    test/test_uniquenessset.py:47:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    /usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
    /usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors) /usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    /usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    stack = []
    onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at 0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
    # Each stack item has four elements:
    # * func: The first operation to perform: os.lstat, os.close or os.rmdir.
    # Walking a directory starts with an os.lstat() to detect symlinks; in
    # this case, func is updated before subsequent operations and passed to
    # onexc() if an error occurs.
    # * dirfd: Open file descriptor, or None if we're processing the top-level
    # directory given to rmtree() and the user didn't supply
    dir_fd.
    # * path: Path of file to operate upon. This is passed to
    onexc() if an
    # error occurs.
    # * orig_entry: os.DirEntry, or None if we're processing the top-level
    # directory given to rmtree(). We used the cached stat() of
    the entry to
    # save a call to os.lstat() when walking subdirectories.
    func, dirfd, path, orig_entry = stack.pop()
    name = path if orig_entry is None else orig_entry.name
    try:
    if func is os.close:
    os.close(dirfd)
    return
    if func is os.rmdir:
    os.rmdir(name, dir_fd=dirfd)
    return

    # Note: To guard against symlink races, we use the standard
    # lstat()/open()/fstat() trick.
    assert func is os.lstat
    E AssertionError
    /usr/lib64/python3.13/shutil.py:663: AssertionError

    What does "mock" do?

    func should be either os.close, os.rmdir or os.lstat.

    If mock is somehow replacing one of those functions, then it might break
    the code.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Left Right@olegsivokon@gmail.com to comp.lang.python on Thu Oct 24 17:17:26 2024
    From Newsgroup: comp.lang.python

    From reading the code where the exception is coming from, this is how
    I interpret the intention of the author: they build a list (not sure
    why they used list, when there's a stack datastructure in Python)
    which they use as a stack, where the elements of the stack are
    4-tuples, the important part about these tuples is that the first
    element is the operation to be performed by rmtree() has to be one of
    the known filesystem-related functions. The code raising the exception
    checks that it's one of those kinds and if it isn't, crashes.
    There is, however, a problem with testing equality (more strictly,
    identity in this case) between functions. I.e. it's possible that a
    function isn't identical to itself is, eg. "os" module was somehow
    loaded twice. I'm not sure if that's a real possibility with how
    Python works... but maybe in some cases, like, multithreaded
    environments it could happen...
    To investigate this, I'd edit the file with the assertion and make it
    print the actual value found in os.lstat and func. My guess is that
    they are both somehow "lstat", but with different memory addresses.
    On Thu, Oct 24, 2024 at 4:06 PM Christian Buhtz via Python-list <python-list@python.org> wrote:

    Hello,
    I am upstream maintainer of "Back In Time" [1] investigating an issue a distro maintainer from Fedora reported [2] to me.

    On one hand Fedora seems to use a tool called "mock" to build packages
    in a chroot environment.
    On the other hand the test suite of "Back In Time" does read and write
    to the real file system.
    One test fails because a temporary directory is cleaned up using shutil.rmtree(). Please see the output below.

    I am not familiar with Fedora and "mock". So I am not able to reproduce
    this on my own.
    It seems the Fedora maintainer also has no clue how to solve it or why
    it happens.

    Can you please have a look (especially at the line "assert func is os.lstat").
    Maybe you have an idea what is the intention behind this error raised by
    an "assert" statement inside "shutil.rmtree()".

    Thanks in advance,
    Christian Buhtz

    [1] -- <https://github.com/bit-team/backintime>
    [2] -- <https://github.com/bit-team/backintime/issues/1911>

    __________________________ General.test_ctor_defaults __________________________
    self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
    """Default values in constructor."""
    with TemporaryDirectory(prefix='bit.') as temp_name:
    test/test_uniquenessset.py:47:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    /usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
    /usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors) /usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    /usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    stack = []
    onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at 0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
    # Each stack item has four elements:
    # * func: The first operation to perform: os.lstat, os.close or os.rmdir.
    # Walking a directory starts with an os.lstat() to detect symlinks; in
    # this case, func is updated before subsequent operations and passed to
    # onexc() if an error occurs.
    # * dirfd: Open file descriptor, or None if we're processing the top-level
    # directory given to rmtree() and the user didn't supply
    dir_fd.
    # * path: Path of file to operate upon. This is passed to
    onexc() if an
    # error occurs.
    # * orig_entry: os.DirEntry, or None if we're processing the top-level
    # directory given to rmtree(). We used the cached stat() of
    the entry to
    # save a call to os.lstat() when walking subdirectories.
    func, dirfd, path, orig_entry = stack.pop()
    name = path if orig_entry is None else orig_entry.name
    try:
    if func is os.close:
    os.close(dirfd)
    return
    if func is os.rmdir:
    os.rmdir(name, dir_fd=dirfd)
    return

    # Note: To guard against symlink races, we use the standard
    # lstat()/open()/fstat() trick.
    assert func is os.lstat
    E AssertionError
    /usr/lib64/python3.13/shutil.py:663: AssertionError

    --
    https://mail.python.org/mailman/listinfo/python-list
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From MRAB@python@mrabarnett.plus.com to comp.lang.python on Thu Oct 24 16:44:30 2024
    From Newsgroup: comp.lang.python

    On 2024-10-24 16:17, Left Right via Python-list wrote:
    From reading the code where the exception is coming from, this is how
    I interpret the intention of the author: they build a list (not sure
    why they used list, when there's a stack datastructure in Python)
    which they use as a stack, where the elements of the stack are
    4-tuples, the important part about these tuples is that the first
    element is the operation to be performed by rmtree() has to be one of
    the known filesystem-related functions. The code raising the exception
    checks that it's one of those kinds and if it isn't, crashes.

    There is, however, a problem with testing equality (more strictly,
    identity in this case) between functions. I.e. it's possible that a
    function isn't identical to itself is, eg. "os" module was somehow
    loaded twice. I'm not sure if that's a real possibility with how
    Python works... but maybe in some cases, like, multithreaded
    environments it could happen...

    To investigate this, I'd edit the file with the assertion and make it
    print the actual value found in os.lstat and func. My guess is that
    they are both somehow "lstat", but with different memory addresses.

    The stack is created on line 760 with os.lstat and entries are appended
    on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or
    an entry is replaced.

    On Thu, Oct 24, 2024 at 4:06 PM Christian Buhtz via Python-list <python-list@python.org> wrote:

    Hello,
    I am upstream maintainer of "Back In Time" [1] investigating an issue a
    distro maintainer from Fedora reported [2] to me.

    On one hand Fedora seems to use a tool called "mock" to build packages
    in a chroot environment.
    On the other hand the test suite of "Back In Time" does read and write
    to the real file system.
    One test fails because a temporary directory is cleaned up using
    shutil.rmtree(). Please see the output below.

    I am not familiar with Fedora and "mock". So I am not able to reproduce
    this on my own.
    It seems the Fedora maintainer also has no clue how to solve it or why
    it happens.

    Can you please have a look (especially at the line "assert func is
    os.lstat").
    Maybe you have an idea what is the intention behind this error raised by
    an "assert" statement inside "shutil.rmtree()".

    Thanks in advance,
    Christian Buhtz

    [1] -- <https://github.com/bit-team/backintime>
    [2] -- <https://github.com/bit-team/backintime/issues/1911>

    __________________________ General.test_ctor_defaults
    __________________________
    self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
    """Default values in constructor."""
    with TemporaryDirectory(prefix='bit.') as temp_name:
    test/test_uniquenessset.py:47:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    /usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
    /usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    /usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    /usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    _ _ _ _
    stack = []
    onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at
    0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
    # Each stack item has four elements:
    # * func: The first operation to perform: os.lstat, os.close or
    os.rmdir.
    # Walking a directory starts with an os.lstat() to detect
    symlinks; in
    # this case, func is updated before subsequent operations and
    passed to
    # onexc() if an error occurs.
    # * dirfd: Open file descriptor, or None if we're processing the
    top-level
    # directory given to rmtree() and the user didn't supply
    dir_fd.
    # * path: Path of file to operate upon. This is passed to
    onexc() if an
    # error occurs.
    # * orig_entry: os.DirEntry, or None if we're processing the
    top-level
    # directory given to rmtree(). We used the cached stat() of
    the entry to
    # save a call to os.lstat() when walking subdirectories.
    func, dirfd, path, orig_entry = stack.pop()
    name = path if orig_entry is None else orig_entry.name
    try:
    if func is os.close:
    os.close(dirfd)
    return
    if func is os.rmdir:
    os.rmdir(name, dir_fd=dirfd)
    return

    # Note: To guard against symlink races, we use the standard
    # lstat()/open()/fstat() trick.
    assert func is os.lstat
    E AssertionError
    /usr/lib64/python3.13/shutil.py:663: AssertionError

    --
    https://mail.python.org/mailman/listinfo/python-list

    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From MRAB@python@mrabarnett.plus.com to comp.lang.python on Thu Oct 24 18:08:02 2024
    From Newsgroup: comp.lang.python

    On 2024-10-24 17:30, Left Right wrote:
    The stack is created on line 760 with os.lstat and entries are appended
    on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or
    an entry is replaced.

    But how do you know this code isn't executed from different threads?
    What I anticipate to be the problem is that the "os" module is
    imported twice, and there are two references to "os.lstat". Normally,
    this wouldn't cause a problem, because they are the same function that doesn't have any state, but once you are trying to compare them, the
    identity test will fail, because those functions were loaded multiple
    times into different memory locations.

    I don't know of any specific mechanism for forcing the interpreter to
    import the same module multiple times, but if that was possible (which
    in principle it is), then it would explain the behavior.
    The stack is a local variable and os.lstat, etc, are pushed and popped
    in one function and then another that it calls, so they're in the same
    thread.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Left Right@olegsivokon@gmail.com to comp.lang.python on Thu Oct 24 18:30:21 2024
    From Newsgroup: comp.lang.python

    The stack is created on line 760 with os.lstat and entries are appended
    on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or
    an entry is replaced.

    But how do you know this code isn't executed from different threads?
    What I anticipate to be the problem is that the "os" module is
    imported twice, and there are two references to "os.lstat". Normally,
    this wouldn't cause a problem, because they are the same function that
    doesn't have any state, but once you are trying to compare them, the
    identity test will fail, because those functions were loaded multiple
    times into different memory locations.

    I don't know of any specific mechanism for forcing the interpreter to
    import the same module multiple times, but if that was possible (which
    in principle it is), then it would explain the behavior.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Left Right@olegsivokon@gmail.com to comp.lang.python on Thu Oct 24 21:21:17 2024
    From Newsgroup: comp.lang.python

    The stack is created on line 760 with os.lstat and entries are appended on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or an entry is replaced.

    But the _rmtree_safe_fd() compares func to a *dynamically* resolved
    reference: os.lstat. If the reference to os changed (or os object was
    modified to have new reference at lstat) between the time os.lstat was
    added to the stack and the time of comparison, then comparison
    would've failed. To illustrate my idea:

    os.lstat = lambda x: x # thread 1
    stack.append((os.lstat, ...)) # thread 1
    os.lstat = lambda x: x # thread 2
    func, *_ = stack.pop() # thread 1
    assert func is os.lstat # thread 1 (failure!)

    The only question is: is it possible to modify os.lstat like that, and
    if so, how?

    Other alternatives include a malfunctioning "is" operator,
    malfunctioning module cache... all those are a lot less likely.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From MRAB@python@mrabarnett.plus.com to comp.lang.python on Thu Oct 24 20:54:53 2024
    From Newsgroup: comp.lang.python

    On 2024-10-24 20:21, Left Right wrote:
    The stack is created on line 760 with os.lstat and entries are appended on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or an entry is replaced.

    But the _rmtree_safe_fd() compares func to a *dynamically* resolved reference: os.lstat. If the reference to os changed (or os object was modified to have new reference at lstat) between the time os.lstat was
    added to the stack and the time of comparison, then comparison
    would've failed. To illustrate my idea:

    os.lstat = lambda x: x # thread 1
    stack.append((os.lstat, ...)) # thread 1
    os.lstat = lambda x: x # thread 2
    func, *_ = stack.pop() # thread 1
    assert func is os.lstat # thread 1 (failure!)

    The only question is: is it possible to modify os.lstat like that, and
    if so, how?

    Other alternatives include a malfunctioning "is" operator,
    malfunctioning module cache... all those are a lot less likely.
    What is the probability of replacing os.lstat, os.close or os.rmdir from another thread at just the right time?
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Left Right@olegsivokon@gmail.com to comp.lang.python on Thu Oct 24 22:08:31 2024
    From Newsgroup: comp.lang.python

    What is the probability of replacing os.lstat, os.close or os.rmdir from another thread at just the right time?

    If the thead does "import os", and its start is logically connected to
    calling _rmtree_safe_fd(), I'd say it's a very good chance! That is,
    again, granted that the reference to os.lstat *can* be modified in
    this way.

    But, before we keep guessing any further, it'd be best if OP could get
    us the info on what's stored in "func" and "os.lstat" at the time the
    assertion fails.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From 2QdxY4RzWzUUiLuE@2QdxY4RzWzUUiLuE@potatochowder.com to comp.lang.python on Thu Oct 24 16:25:56 2024
    From Newsgroup: comp.lang.python

    On 2024-10-24 at 20:54:53 +0100,
    MRAB via Python-list <python-list@python.org> wrote:

    On 2024-10-24 20:21, Left Right wrote:
    The stack is created on line 760 with os.lstat and entries are appended
    on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

    'func' is popped off the stack on line 651 and check in the following lines.

    I can't see anywhere else where something else is put onto the stack or
    an entry is replaced.

    But the _rmtree_safe_fd() compares func to a *dynamically* resolved reference: os.lstat. If the reference to os changed (or os object was modified to have new reference at lstat) between the time os.lstat was added to the stack and the time of comparison, then comparison
    would've failed. To illustrate my idea:

    os.lstat = lambda x: x # thread 1
    stack.append((os.lstat, ...)) # thread 1
    os.lstat = lambda x: x # thread 2
    func, *_ = stack.pop() # thread 1
    assert func is os.lstat # thread 1 (failure!)

    The only question is: is it possible to modify os.lstat like that, and
    if so, how?

    Other alternatives include a malfunctioning "is" operator,
    malfunctioning module cache... all those are a lot less likely.
    What is the probability of replacing os.lstat, os.close or os.rmdir from another thread at just the right time?

    That is never the right question in a multi-threaded system. The answer
    is always that is doesn't matter, the odds will beat you in the end. Or sometimes right in the middle of a CPU instruction; does anyone remember
    the MC680XX series?

    Yes, as a matter of fact, I did used to make my living designing,
    building, delivering, and maintaining such systems.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Barry@barry@barrys-emacs.org to comp.lang.python on Thu Oct 24 23:44:43 2024
    From Newsgroup: comp.lang.python


    On 24 Oct 2024, at 15:07, Christian Buhtz via Python-list <python-list@python.org> wrote:

    On one hand Fedora seems to use a tool called "mock" to build packages in a chroot environment.
    On the other hand the test suite of "Back In Time" does read and write to the real file system.
    I am a Fedora packager and can help explain what is the tools are doing.
    Mock runs the build in a chroot env that allows for reproducible clean room builds.
    Sort like a container.
    This is nothing to do with the python mock package.
    What do you mean by the real file sustem?
    You cannot write to the /usr file system. Is that what your tests do?
    If so that needs changing.
    Barry
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From c.buhtz@c.buhtz@posteo.jp to comp.lang.python on Fri Oct 25 06:59:28 2024
    From Newsgroup: comp.lang.python

    Thank you very much for all your responses.

    Am 24.10.2024 17:17 schrieb Left Right:
    To investigate this, I'd edit the file with the assertion and make it
    print the actual value found in os.lstat and func. My guess is that
    they are both somehow "lstat", but with different memory addresses.

    My reporter provided this [1]. I think this is the relevant output:

    =========================== short test summary info ============================
    FAILED test/test_plugin_usercallback.py::SystemTest::test_local_snapshot
    - As...
    FAILED test/test_uniquenessset.py::General::test_ctor_defaults - AssertionError
    FAILED test/test_uniquenessset.py::General::test_deep_check -
    AssertionError
    FAILED
    test/test_uniquenessset.py::General::test_fail_equal_without_equal_to
    FAILED test/test_uniquenessset.py::General::test_size_mtime -
    AssertionError
    FAILED test/test_uniquenessset.py::General::test_unique_myself - AssertionError
    FAILED test/test_uniquenessset.py::General::test_unique_size_but_different_mtime ================== 7 failed, 267 passed, 16 skipped in 20.79s ==================
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    os.lstat=<built-in function lstat>
    make: Leaving directory '/home/johannes/rpmbuild/BUILD/backintime-1.5.3-build/backintime-1.5.3-rc1/common'

    RPM build errors:



    [1] -- <https://github.com/bit-team/backintime/issues/1911#issuecomment-2436851901>



    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From c.buhtz@c.buhtz@posteo.jp to comp.lang.python on Fri Oct 25 07:06:35 2024
    From Newsgroup: comp.lang.python

    Hello Barry,

    thank you for your reply and clarifying the Fedora aspects.

    Am 25.10.2024 00:44 schrieb Barry:
    What do you mean by the real file sustem?

    You cannot write to the /usr file system. Is that what your tests do?
    If so that needs changing.

    Asking the right questions brings up to important details. While I was
    writing and trying to explain that the relevant test does use ""tempfile.TemporaryDirectory" as a context, I realized that "PyFakeFS"
    is used in the back [1].

    But that makes me wonder. On a "regular" system all tests are running.
    So the issue might exist because of a combination of 3 factors: shutil.rmtree(), PyFakeFS in a chroot/mock build environment.

    [1] -- <https://github.com/bit-team/backintime/blob/c1d042ab67b9e117ac53e944518a0f4292fa075b/common/test/test_uniquenessset.py#L45>
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From c.buhtz@c.buhtz@posteo.jp to comp.lang.python on Fri Oct 25 07:29:19 2024
    From Newsgroup: comp.lang.python

    Am 25.10.2024 09:06 schrieb Christian Buhtz via Python-list:
    On a "regular" system all tests are running.

    To clarify: "regular" does not exclude PyFakeFS. It means on my own
    local development machine and on the TravsCI machines (Ubuntu 22 with
    Python 3.9 up to 3.13) and using PyFakeFS in that test, everything is
    fine.
    Only when mock/chroot is involved that happens.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From c.buhtz@c.buhtz@posteo.jp to comp.lang.python on Sat Oct 26 11:08:05 2024
    From Newsgroup: comp.lang.python

    As you can see in the linked issue it seems it was an incompatibility
    between the version of Python and PyFakeFS.

    In the end it was a Fedora packaging bug because that pyfakefs version
    was not compatible with Python 3.13.

    Thanks in advance for helping out.
    --- Synchronet 3.20a-Linux NewsLink 1.114
  • From Barry@barry@barrys-emacs.org to comp.lang.python on Sun Oct 27 08:56:59 2024
    From Newsgroup: comp.lang.python


    On 26 Oct 2024, at 12:11, Christian Buhtz via Python-list <python-list@python.org> wrote:

    As you can see in the linked issue it seems it was an incompatibility between the version of Python and PyFakeFS.

    In the end it was a Fedora packaging bug because that pyfakefs version
    was not compatible with Python 3.13.
    That makes sense.

    Thanks in advance for helping out.
    No problem.
    Barry
    --
    https://mail.python.org/mailman/listinfo/python-list

    --- Synchronet 3.20a-Linux NewsLink 1.114