Hello,
I'm living in Almaty, so I won't be great help for you in Atyrau, however I'll give you some information.
There are good gyms in the city. You have the choice between the most expensive ones : Fidelity, a huge gym center in the district of Samal, the one at the Hyatt and the one at the Intercontinental. The three of them are brand new and mostly used by expats.
Otherwise, you can try local gyms, which I've been told are not bat at all, and less expensive.
Kazakh people like sports very much and besides gym you can enjoy karate, tennis, golf, dance and yoga lessons e.g. at independent clubs.
As for me, I'm doing ayangar yoga at a private club with a Russian teacher.