I was able to isolate the issue, when using a dual ipv4 + ipv6 SmallWall, if you have the traffic shaper turned on with any rules that touch any ipv6 traffic on the "Outbound", aka, Upload, then this issue happens. Basically ipv6 only works for a minute or two until some traffic has to filter through those rules, then it gets stopped or packet loss. If you turn off the traffic shaper it will solve that issue, but then you have no traffic shaper. Download rules seem to work fine (Inbound Traffic) and if you make an Upload rule that *only* applies to ipv4 traffic, that will work to. Basically, can't have any rules to apply against Outbound (Upload) ipv6 traffic.
I have not seen it but there has also been very little testing with both v4 and v6 IPs.
I actually remember this problem back in the m0n0wall days, but at the time didn't know that what it was. I had setup a site when ipv6 was still kind of *new* and had the same issue of sparatic working. I thought it was an ISP issue and just turned off the ipv6 since they didn't want it at that time. Years later, , the ghost has returned, but at least now I know why it happens instead of just shrugging my shoulders and turning off ipv6.