For a standard piston based engine, the red line is mainly decided by the speed at which the engine starts to shake itself to pieces due to vibration caused by small imbalances in the weights of the pistons, and so on. As rotational speed goes up, the weight imbalance would increase exponentially, if my basic Physics serves me right. Race cars with higher than normal redlines have very precisely machined and balanced parts, which means they can handle much higher rotational speeds without shaking themselves to pieces.

I'm quite sure the response time of spark plugs and fuel injectors also come into play at very high engine speeds, too.

A rotary engine, due to its simple, rotational by nature design, can just rev and rev and rev liek there's no tomorrow. But I am sure that perfectly balanced rotors could help push the red line even higher...

 

^^^^ even higher redlines?... mmmm fun thoughts