Your decision to marry him was based on a promise but not on the reality of where he is religiously. Despite not being religiously observant, he is encouraging you to grow religiously and does not stand in your way of spiritual growth. His relationship with Allah, subhanahu wa ta`ala (exalted is He), is a personal one and despite your well-intentioned encouragement, he may only be hearing nagging and judgment from you. He may not feel accepted by you for who he is despite his flaws, so he will continue to pull away from you because you may remind him to feel awful about himself for not praying dutifully.Approach your family with tenderness and understanding as they progress in the faith and continue to rely on Allah (swt) when you have feelings of fear and anxiety.