Go to Settings -> Teams -> Edit the team

Expand Team Info and make sure the option Apply selection change of team members to the existing calendars which got linked to this team is toggled off

That way when we add a new team member, it will not update all the calendar team members associated with this team.