"Magic Markers" on porous surfaces like your wood, are almost impossible to remove with solvents.
As you said, the solvent may dissolve the marker, but it also drives it deeper into the wood, etc. If lighter fluid or alcohol don't work, you might try acetone or methyl ethyl ketone (MEK).
WD-40 also works sometimes on some surfaces, but then you have to remove it with lighter fluid, etc, and like the others it soaks into porous surfaces.
The last resort, that usually works on wood, is sandpaper, but only if the marker hasn't soaked so deep that sanding won't get it.
On fabric, you can place a paper towel behind the spot to catch the solvent as it soaks thru. This helps prevent the spot from just getting larger as the solvent/dye absorbs into the fabric.