Absolutely. The yellow card scheme is there for an absolutely critical reason and it MUST be used properly to be effective.
The entire point of the scheme is to spot events even if they seem like red herrings when they are seen in isolation.