They matter a great deal in Heat Recovery Units and for the entire Heat Recovery System.
Filters ensure that air being drawn in from outside the building is suitably filtered of any airborne particles such as dust and pollen before if enters the home. In families where there are allergies and conditions such as asthma this is a crucial part of well-being in a home with heat recovery ventilation.
If filters are clogged, the heat recovery unit won't be able to draw in air at the rate it is supposed to (just like your clogged hoover filter!). This will then impact airflow rates (both supply and extract), it may introduce noise as the system will be working harder and overall, it will impact the comfort of the home.