This adds a pre-filter that drops out-of-window non-recurring events
from the raw ICS data before `node-ical` parses it. Recurring events and
anything we cannot classify cheaply are kept - the existing logic
handles those as before.
Some calendars accumulate years of old events. Pre-filtering them
reduces the amount of work `node-ical` has to do and keeps event loop
lag lower. My benchmarks on a fast machine showed a small but measurable
speedup (roughly 10-20% with generated test data); on a Raspberry Pi it
should be significantly more noticeable.
For the implementation I ended up using
[`ics-filter`](https://github.com/runely/ics-filter), suggested by
@rejas. I had a custom version first, but ICS date string parsing has
enough complexity that it makes sense to delegate it to a package built
specifically for this. I also contributed a couple of fixes to
`ics-filter` along the way, and the maintainer was responsive and open
to improvements, so it seems like a good fit.
Solves #4103.
When a server responds with 304 (nothing changed since last fetch), the
response has no body. Several modules were trying to parse that empty
body anyway - which either cleared their cached data or threw an
exception. The result: a blank calendar, empty newsfeed, or missing
weather data after the next refresh cycle.
This was reported in the forum:
https://forum.magicmirror.builders/topic/20250/calendar-events-broadcasting-nothing-showing
The bug was "introduced" by #4120, which correctly started forwarding
304s to consumers - but not all were ready for it.
### Fix
Skip parsing on 304 and keep the existing data as-is:
- **calendar** - re-broadcasts cached events
- **newsfeed** - re-broadcasts cached items
- **buienradar, openmeteo, weatherflow, weathergov** - return early
before calling `response.json()`
This switches the calendar fetcher from synchronous to asynchronous ICS
parsing.
It does not necessarily make parsing faster overall, but it avoids long
event-loop stalls with large calendar files (especially on slower
devices and with multiple feeds).
So this does not fully solve #4103 on its own, but it clearly mitigates
it.
It should also combine well with a future pre-filter approach discussed
in the issue:
- with pre-filter = less data to parse (future PR)
- with async parsing = less blocking while parsing (this PR)
Together, that is likely the strongest path to fully address #4103.
In PR #4072 GitHub Bot complained about an unused var. Instead of just
removing that one, I checked why ESLint hadn't complained about it: We
had disabled the rule for it.
So I enabled rule and resolved the issues that ESLint then detected.
Related to #4073
Since the project's inception, I've missed a clear separation between
default and third-party modules.
This increases complexity within the project (exclude `modules`, but not
`modules/default`), but the mixed use is particularly problematic in
Docker setups.
Therefore, with this pull request, I'm moving the default modules to a
different directory.
~~I've chosen `default/modules`, but I'm not bothered about it;
`defaultmodules` or something similar would work just as well.~~
Changed to `defaultmodules`.
Let me know if there's a majority in favor of this change.
Kristjan ESPERANTO
Karsten Hassel