In addition to what K447 said, you could measure the current that the LEDs use on a fresh set of batteries and then in order to use a 12V adapter, calculate what value resistance to stick in series.
R=voltage/current. Assuming the LEDs draw 40mA and you want to drop 6V (12-6) R=6/0.04= 150Ohm.
Power dropped over the resistor would be P=UxI=6*0,04=0,24W.
The most commonly available resistor type is 1/4W so it would technically do, but it would be running hot, close to its limit. Better to use two or thee resistors in series or parallel to spread the load across them. Maybe 3 51 Ohm resistors in series, or 3 470 Ohm in parallel. Don't bunch them up, give them some distance between them and they should run comfortably.

Another way to drop the voltage would be to stick a number of diodes (maybe 1N4001 ones) in series, each dropping typically around 0.6-0.7V depending on the current. Maybe get a 10-pack, stick them in all in series and check the result. If too dim, remove a diode and check again. Repeat if necessary.
That's what I found inside a cig.lighter adapter for a portable (4xAA) CD player a loong time ago... Crude but worked despite the varying supply voltage of the car (about 12-14.4V) since the player was designed to accept quite a bit of variation in voltage from the batteries it normally ran off.
Or an actual voltage regulator like a 7806 or a LM317 (with a couple resistors to set its voltage.

Unless the lights have some active electronics to regulate the current, the brightness will vary some as the batteries drain (I got a few strings of LEDs for Xmas decorations that run of 3xAA, so nominally 4.5V) But they also work just fine off a 18650 LiIon rechargeable that fully charged is 4.2 and for most of the run is about 3.7V Once I notice pretty much any dimming it means the battery is starting to drop down the tail of the curve and it's time to recharge/switch the battery. Plain AAs I'd run much further down as they're disposable, but LiIon rechargeables don't like being discharged too deep.

If you don't have a 6V adapter and don't want to play with any of the above, you might well find that the light runs at an acceptable level off a 5V adapter, which these days are extremely common and easy to find.