This is aninterest of mine: passively attaching tags. I've long-thought weed burrs, perhaps with a bit of glue, could be used to attach devices. I am super keen to try out methods if you and others are interested.
From elsewhere, I know there are plans afoot to design passive attachment tags for polar bears: https://polarbearsinternational.org/news/article-research/qa-burr-on-fur/
And years ago somebody wrote an article about 'self-attaching' wolf (I think) collars, that were sort of like a snare that broke away once it was around a wolf's neck. I can't find the paper but if I do I'll send a link.
There's also this: https://wildlife.onlinelibrary.wiley.com/doi/abs/10.1002/wsb.1034
If you let me know an email address I can send you the article. It's a crazy, but amazing set up.
There are a few more things around, but suprisingly, this is an area needing a lot more attention I think.
In terms of cheap-ish store-on-board GPS, a popular unit is the i-gotU GT-120, but I think they are hard to get as they are a fairly old unit. I think (but am not sure) that these are very similar, so try here maybe: http://www.mr-lee-catcam.de/pe_cc_i9.htm (see the Shop page).
There's also the GT-600 (which I haven't used before), but at least looks like you can still buy (e.g. https://canadagps.ca/products/i-gotu-gt-600-usb-gps-travel-sports-logger...). It has Bluetooth if that's of any use...Usually, people hack the case open on i-gotUs and add a bigger battery (but I've only ever been able to get larger capacity lipo packs to work). But, if you only want relatively short deployments, you could probably leave it as is. You might have to make sure the top switch can't be accidentally triggered though.
There are lots of other GPS systems around, but most are more expensive.
Happy to chat more about any of this...