The なければ is just a regular conditional. “If she didn’t keep screaming”.
気を失う is an expression, it means “to lose consciousness”.
The のあまり part when attached to something gives the idea that something is too strong and leads to another thing happening. In this case, she was so scared that, if she didn’t keep screaming, she would have fainted.