"when it gets hard to breathe"
Yes, this bit of spectacularly unhelpful advice is pretty typical of the safety literature and mask owners manuals...30 years ago. But without knowing how many particles are in your air, how much air you use with each lung full, and how much breathing restriction you can tolerate, it's really about all anyone's going to be able to tell you over the phone.
"Hard to breathe" is both relative depending on a person's lung health and difficult to determine because your filters don't clog all at once..it just gets gradually harder to breathe. But in your situation, "hard to breathe" is somewhat irrelevant, unless you are doing a really dusty job. You should probably be replacing your filters before they are clogged for the sake of hygiene. The humid air will moisten the filter material when you inhale, and mold will flourish if either the filter material or dust are a food source. If your inhalation valves at the filters are leaky when you exhale (do you know how to test these?) humidity/saliva will end up on your side of the filter material and support mold/bacteria.
How often to replace the filters...I remember living in Florida, you could practically watch mildew grow. If it were me and I was wearing this mask 8 hours a day, I would probably replace the filters daily. Actually if I were wearing it 8 hours a day I would get a powered respirator, these come with a little tester that will tell you when the batteries are weak or the filters are clogged. And they are much cooler and easier to breathe through. If I was only wearing a respirator an hour a day I might be tempted to find some sort of storage that would not let the mold grow in the filters, but experimenting with your health to save a few bucks sometimes does not work out well.