Sudden higher or minimal temperatures might create the glass to warp and this could guide to gaps via which air leaks.
In summer time, consider down large curtains and valances and replace them with lighter types.