false alarm

A thing or occurrence which initially causes fear, distress, etc. but which is subsequently recognized as being no cause for concern.