Just read your agreement one more time, to make sure it is what it is.
If your idea is not work related. The general rule says you are safe.
I also keep an inventor log book that can prove that all my ideas are forming at my spare time (not at work). At least I log them at my spare time since I keep it at home.
Counsel lawyers and they will tell you pretty much the same thing except they will go through the every fine prints more detail than you and I. If you can do the same, you are fine. The most important thing what a lawyer tell you is just an valid opinion, not a guarantee. You can always use your own judgment to do the same.