To answer your questions:
>> 1. Will the device even accept that kind of battery?
The Audiomoth comes with a built in AAx3 alkaline battery holder. You would need to solder some leads to the (+) and (-) side of that battery holder to connect to a lithium-ion cell. In terms of voltage compatibility, according to the Audiomoth schematics v1.1.0 which is published, it should be no problem, although it's a bit tight. The system runs at 3.3V and as long as you have a voltage supply at least 0.3V above that, things should run fine. The nominal voltage of a lithium ion battery is 3.7V so it's just within that threshold. You may not "officially" be able to use the full capacity of a lithium-ion battery though as it would drop down to 3.0V which is too low for the regulator. However it's likely that things will probably function through most of the battery's life.
>> 2. How long do they last compared to NiMH rechargeable or regular lithium ion batteries.
@htarold gave a good explanation above, but one caveat is that you may not be able to use the full life of the battery depending how the system behaves once the battery voltage droops below 3.6V. you can double or triple up lithium-ion batteries though which would increase the battery life.
>> 3. What brands would you recommend? It doesn't seem like any of the "common" brands (Duracell, Energizer, etc.) are producing these for consumer purposes.
Again as @htarold mentioned, Lithium-ion batteries are harder to come by than akalines. They require proper handling and protection, otherwise, they can chain-react and quickly release their stored energy. Those were the hoverboard fires and Samsung mobile phone fires that made the news a few years back. That said, you can purchase lithium-ion batteries with protection circuitry or battery holders that incorporate protection circuitry to prevent any errant behavior. Jacinta and I are working on battery modules like this for our field designs because we need to be able to charge them via solar panels.
I'd recommend playing with them to start getting familiar with lithium-ion batteries. I haven't had one explode into flames on me and I use them alot. Usually I have the opposite scenario. The batteries get drained too much where they get damaged and can no longer be charged. There are ways to revive them, but this is also one of the reasons you need protection on them. They need to be protected from undervoltage (<3.0V), overvoltage (>4.2V), and overcurrent (enough current where they start heating up uncontrollably).
Hope that helps.