Swamp Stalkers need to have access to Water - either naturally in the terrain or via minor terrain - to power their Born of the Swamp ability. If they can't find water, they can get decimated with low saves.
Their magic, missile, and light melee units almost all bear a "wasted" face, which makes them less reliable than they could otherwise be.
Their large cavalry unit trades the more common Trample, Fly, or Rend SAIs for the Swamp Stalker exclusive SAI Coil. While it isn't a terrible SAI, it is risky and leaves the possibility for the coiling unit to be killed, and it hinders the usefulness of an otherwise great die.
Most of their melee units don't hit particularly hard, except the heavy melee units which bear no maneuver icons at all. This means most strong melee options require cavalry to get them moving, which weakens their attack damage.
Their mages lack strong icon modifiers. Without spells like Transmute Rock to Mud, Fiery Weapon, or Wind Walk, they are limited to natural icons rather than being able to modify the outcomes directly.