Do other emulators allow pressing both left/right at the same time. If I remember correctly, this is the behavior you'd expect in this game.

Many emulators have special handling of what happens if you press opposing directions on the d-pad at the same time. If you allow both keypresses funny things can happen, including things like Metal Slug Advance crashing, and an assortment of other glitches. It may just be that the different emulators have different handling of this. There's a setting for how to handle it in mGBA too.

I'm pretty sure that THIS was not supposed to happen.

You can't press them at the same time on hardware. There is no "supposed to".

That's what I'm talking about.
This is a glitch caused by using the keyboard.

Do you have "Allow opposing input directions" checked in Settings > Interface?

I have a broken dpad on one of my GBAs I wonder if I could do this in the game.

"Allow opposing input directions" was disabled
I enabled and it's still the happens.

Oh. Looks like that feature broke then.

I tested on 07-b1 and the glitch still happens but on 0.6.3 was fine.