The size of the propeller that can be used is determined by the frame size. The maximum size of propeller that can be used is the minimum frame dimension of length or width divided by two.
Figure 3 shows a square drone frame configuration, so the maximum propeller size is X/2.
Figure 4 shows a rectangular frame size so in this case, Y is the obvious minimum dimension, so the maximum propeller size is Y/2.
Overlapping the propellers is not recommended since the downward airflow of one propeller will reduce the thrust efficiency of the propeller below. There is also a bit of side to side airflow due to tip vortices, so you want to allow for some spacing between the propellers to avoid this effect.
Fig 3. Frame and Prop Size
Fig 4. Minimum Frame and Prop Size