BUG: Copy not working as expected within DataFrameGroupBy.apply function

[dakenblack]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

  • present in v1.2.3

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A' : np.random.normal(10, 12, size=(100, ))
})

# function to cyclically group with a frequency of 4
def grouper(index):
    return index % 4

store = {}

# debuging apply function to store the dataframe chunks created by the grouper
# also contains print statements to identify the error
def addstore(df):
    key = df.index.min()
    
    if len(store.keys()) > 0:
        firstkey = list(store.keys())[0]
        print(f'{key:19d} {firstkey:19d} {store[firstkey].index.min():19d} {firstkey == store[firstkey].index.min()!s:6s}')
    else:
        print('New key', key)
        print(f'{"New Key":19s} {"First Key":19s} {"Stored Index":19s} {"Valid":19s}')
        
    # attempting to copy dataframe
    store[key] = pd.DataFrame(df).copy(deep=True)

df.groupby(grouper).apply(addstore)

A more detailed example with integer and datetime indices can be found here: https://gist.github.com/dakenblack/41595c831def6d06e289f87d015aca5e

Problem description

Between calls of the apply function (getstore) the copied dataframe changes. If a copy is created, the copy should be independant of the original dataframe.

Found Output

New key 0
New Key             First Key           Stored Index        Valid              
                  1                   0                   1 False 
                  2                   0                   2 False 
                  3                   0                   3 False 

Expected Output

New key 0
New Key             First Key           Stored Index        Valid              
                  1                   0                   0 True 
                  2                   0                   0 True 
                  3                   0                   0 True 

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f2c8480
python : 3.7.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.21343
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.2.3
numpy : 1.18.5
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 49.2.0.post20200714
Cython : 0.29.21
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.3.3
lxml.etree : 4.5.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.20.0
pandas_datareader: None
bs4 : 4.8.0
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : 1.3.13
tables : None
tabulate : 0.8.9
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
numba : None

[phofl]

Hi, thanks for your report. Could you trim down your example? See https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[dakenblack]

Hi, I'm not sure what else to remove from the example. I guess I could remove the print statements. I feel everything else is required to explain the problem:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A' : np.random.normal(10, 12, size=(100, ))
})

# function to cyclically group with a frequency of 4
def grouper(index):
    return index % 4

store = {}

# debuging apply function to store the dataframe chunks created by the grouper
# also contains print statements to identify the error
def addstore(df):
    key = df.index.min()
    
    if len(store.keys()) > 0:
        firstkey = list(store.keys())[0]
        stored_index = store[firstkey].index.min()

        # This is the main problem, the copy (stored_index) seems to change between calls of addstore
        # the stored_index should equal 0 each time but changes with "key" which is the new key found for each group
        print(f'{key:10d} {firstkey:10d} {stored_index:10d}')
        
    # attempting to copy dataframe
    
    store[key] = pd.DataFrame(df).copy(deep=True)

df.groupby(grouper).apply(addstore)

[phofl]

You are creating a new df, which has index 0, do it like this:

df = pd.DataFrame({
    'A': np.random.normal(10, 12, size=(100,))
})


# function to cyclically group with a frequency of 4
def grouper(index):
    return index % 4


store = {}


# debuging apply function to store the dataframe chunks created by the grouper
# also contains print statements to identify the error
def addstore(df):
    key = df.index.min()

    if len(store.keys()) > 0:
        firstkey = list(store.keys())[0]
        stored_index = store[firstkey].index.min()

        # This is the main problem, the copy (stored_index) seems to change between calls of addstore
        # the stored_index should equal 0 each time but changes with "key" which is the new key found for each group
        print(f'{key:10d} {firstkey:10d} {stored_index:10d}')

    # attempting to copy dataframe

    store[key] = df.copy(deep=True)


df.groupby(grouper).apply(addstore)

And you get the expected result

[dakenblack]

I copied your code and the stored_index is still changing. I think the only difference in your code is that you removed the pd.DataFrame around df before calling copy (unless I've missed something).

I tried it this way first of course but the copy was still changing and then put the pd.DataFrame around it thinking it would call the copy constructor or something.

[phofl]

Now I get your issue.

Please try formulating your examples like this in the future:

df = pd.DataFrame({
    'A': np.random.normal(10, 12, size=(4,)),
    "B": [1, 2, 1, 2],
})

store = {}

def addstore(x):
    store[len(store)] = x.copy()

df.groupby("B").apply(addstore)

This should provide a bit more clarity, since there is not that much noise.

which returns

{0:            A  B
1   9.535755  1
3  10.853170  1, 1:            A  B
1  -2.217803  2
3 -18.045611  2}

This is obviously wrong, you are correct.

Interestingly enough, if we add a a view on the underlying object to the dict instead of a copy, this works:

Changing store[len(store)] = x.copy() with store[len(store)] = x[:] returns the desired result.

{0:            A  B
0  10.133216  1
2  15.768876  1, 1:           A  B
1 -9.502439  2
3  5.841476  2}

There seems to be something seriously off with the assignment, @rhshadrach are you able to help here? This happens probably somewhere in apply_frame_axis0, but I am not familiar with the apply implementation. Should this even work? Seems odd to modify an external object from within apply...