From apm at one.com  Thu Jul  3 20:28:48 2008
From: apm at one.com (Peter Mogensen)
Date: Thu, 03 Jul 2008 22:28:48 +0200
Subject: [Calendarserver-maintainers] Bug#489188: Events more than 356 days
	from creation are ignored
Message-ID: <486D3680.50809@one.com>

Package: calendarserver
Version: 1.2.dfsg-2
Severity: important

I've noticed that DCS (1.2) will not return an event instance more than
356 days from the creation of the event in a calendar-query.

It seems to be related to Trac ticket #207 (except that it affects
non-recurring events too)
http://trac.calendarserver.org/ticket/207

I've found that these events are not put into the TIMESPAN index, which
means that they are not returned by a search using the index.

Searches seems to not use the index when there's a time-range element in
the filter, but the exception detecting this is only thrown from the
filter expression generator for prop-filters, ... which again means that
if you don't have a prop-filter in your query, you'll get a search using
the index, which will ignore these 356+ days events.

The solution seems to be in the roadmap for DCS 1.3, however I would
regard this as a major flaw for 1.2 users.

Depending on the exact scenario there's different small patches which
would make these events appear. The simple solution would be to disable
indexed queries, but that would be a major performance hit for busy
servers.

Peter




From noreply at release.debian.org  Mon Jul  7 16:39:16 2008
From: noreply at release.debian.org (Debian testing watch)
Date: Mon, 07 Jul 2008 16:39:16 +0000
Subject: [Calendarserver-maintainers] calendarserver 1.2.dfsg-4 MIGRATED to
	testing
Message-ID: <E1KFtkC-0002nV-RQ@ries.debian.org>

FYI: The status of the calendarserver source package
in Debian's testing distribution has changed.

  Previous version: 1.2.dfsg-1
  Current version:  1.2.dfsg-4

-- 
This email is automatically generated; the Debian Release Team
<debian-release at lists.debian.org> is responsible.
See http://release.debian.org/testing-watch/ for more information.


From noreply at release.debian.org  Mon Jul  7 16:39:18 2008
From: noreply at release.debian.org (Debian testing watch)
Date: Mon, 07 Jul 2008 16:39:18 +0000
Subject: [Calendarserver-maintainers] twisted-calendarserver
	0.2.0.svn19773-5 MIGRATED to testing
Message-ID: <E1KFtkE-0002qS-8S@ries.debian.org>

FYI: The status of the twisted-calendarserver source package
in Debian's testing distribution has changed.

  Previous version: 0.2.0.svn19773-2
  Current version:  0.2.0.svn19773-5

-- 
This email is automatically generated; the Debian Release Team
<debian-release at lists.debian.org> is responsible.
See http://release.debian.org/testing-watch/ for more information.


From owner at bugs.debian.org  Thu Jul 24 15:09:12 2008
From: owner at bugs.debian.org (Debian Bug Tracking System)
Date: Thu, 24 Jul 2008 15:09:12 +0000
Subject: [Calendarserver-maintainers] Processed: bug 489188 is forwarded to
	http://trac.calendarserver.org/ticket/207
In-Reply-To: <1216879136-4135-bts-agx@sigxcpu.org>
References: <1216879136-4135-bts-agx@sigxcpu.org>
Message-ID: <handler.s.C.121691206029234.transcript@bugs.debian.org>

Processing commands for control at bugs.debian.org:

> # Automatically generated email from bts, devscripts version 2.10.34
> forwarded 489188 http://trac.calendarserver.org/ticket/207
Bug#489188: Events more than 356 days from creation are ignored
Noted your statement that Bug has been forwarded to http://trac.calendarserver.org/ticket/207.

>
End of message, stopping processing here.

Please contact me if you need assistance.

Debian bug tracking system administrator
(administrator, Debian Bugs database)



From agx at sigxcpu.org  Thu Jul 24 06:07:49 2008
From: agx at sigxcpu.org (Guido =?UTF-8?Q?G=C3=BCnther?=)
Date: Thu, 24 Jul 2008 02:07:49 -0400
Subject: [Calendarserver-maintainers] Bug#489188: Bug#489188: Events more
	than 356 days from creation are ignored
In-Reply-To: <486D3680.50809@one.com>
References: <486D3680.50809@one.com>
Message-ID: <20080724060749.GB21335@bogon.ms20.nix>

Hi Peter,
sorry for the late reply. Thanks for the detailed report.

On Thu, Jul 03, 2008 at 10:28:48PM +0200, Peter Mogensen wrote:
> It seems to be related to Trac ticket #207 (except that it affects
> non-recurring events too)
> http://trac.calendarserver.org/ticket/207
> 
> I've found that these events are not put into the TIMESPAN index, which
> means that they are not returned by a search using the index.
> 
> Searches seems to not use the index when there's a time-range element in
> the filter, but the exception detecting this is only thrown from the
> filter expression generator for prop-filters, ... which again means that
> if you don't have a prop-filter in your query, you'll get a search using
> the index, which will ignore these 356+ days events.
Could you work on a fix? It seems you've got most of the details
together already. I attached your analysis to Trac ticket #287.

> The solution seems to be in the roadmap for DCS 1.3, however I would
> regard this as a major flaw for 1.2 users.
It certainly is.
 -- Guido




From apm at one.com  Thu Jul 31 10:22:34 2008
From: apm at one.com (Peter Mogensen)
Date: Thu, 31 Jul 2008 12:22:34 +0200
Subject: [Calendarserver-maintainers] Bug#489188: Bug#489188: Events more
	than 356 days from creation are ignored
In-Reply-To: <20080724060749.GB21335@bogon.ms20.nix>
References: <486D3680.50809@one.com> <20080724060749.GB21335@bogon.ms20.nix>
Message-ID: <4891926A.10903@one.com>

Guido G?nther wrote:
>> Searches seems to not use the index when there's a time-range element in
>> the filter, but the exception detecting this is only thrown from the
>> filter expression generator for prop-filters, ... which again means that
>> if you don't have a prop-filter in your query, you'll get a search using
>> the index, which will ignore these 356+ days events.
> Could you work on a fix? It seems you've got most of the details
> together already. I attached your analysis to Trac ticket #287.

I don't have any "perfect" fix. That would be a more complex patch and 
would have to take into account upcoming changes for 1.3.

Here's some more analysis and suggestions for work-arounds, though:

According to RFC4791 a simple calendar-query REPORT would look like this:
================
<calendar-query xmlns:D="DAV:" xmlns="urn:ietf:params:xml:ns:caldav">
   <D:prop>
     <calendar-data/>
   </D:prop>
   <filter>
     <comp-filter name="VCALENDAR">
       <comp-filter name="VEVENT">
         <time-range start="20070420T120039Z" end="20070620T120039Z"/>
       </comp-filter>
     </comp-filter>
   </filter>
</calendar-query>
=================

The "time-range" element is almost in all use cases part of a 
comp-filter. (It could also be used in a prop-filter.)

When an event is inserted into the database its instances are calculated 
  up to 356 days into the future. Any such instances are put into the 
TIMESPAN table of the index. If this is not a complete expansion 
RECURRANCE_MAX is set in the RESOURCE table to indicate how far we have 
indexed this event.
As wsanchez says in Apples Trac ticket 207:

"The index is supposed to indicate how far out expansion has happened 
for each event, and if a query against the index goes beyond that time, 
the expansion should be brought out farther, up to a defined limit.

If we have expanded to the limit, then any queries that goes beyond the 
limit would have to read the icalendar data to see if that event matches 
(far more expensive, but the idea is that such queries should be rare)."

However... RECURRANCE_MAX is never used in the DCS 1.2 code as a quick 
grep of the source will show!!

So no where is the expansion ever brought out farther for events beyond 
356 days from creation!
This means that searches using the index will never see such events - 
even though they are stored correctly in the database.

Searches are using the index if index_query_ok becomes true in
twistedcaldav/method/report_calquery.py which it does when 
index.searchValid is true which in turn is the case when 
query.calendarquery.sqlcalendarquery does not encounter a ValueError.

Now, what to do about it?

In the DCS v/1.2 source in twistedcaldav/query/calendarquery.py line 159 
a ValueError exception is raised since we don't handle time-range within 
prop-filter. Obviously (as described above) we don't correctly handle 
time-range within comp-filter, but the equivalent exception is not 
raised in line 102-105.
On fix would be to completely disable use of the index by just raising a 
ValueError in calendarquery.py by replacing line 104, 105 with
"raise ValueError".
This disables the whole TIMESPAN index.

Another not as brutal, but equally simple fix would be to take 
RECURRANCE_MAX into account, but modifying the SQL qualifier which 
decides which events to include in the search result which is defined in 
twistedcaldav/query/sqlgenerator.py line 49:

Changing:
TIMESPANTEST  = "((TIMESPAN.FLOAT == 'N' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s) OR (TIMESPAN.FLOAT == 'Y' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s)) AND TIMESPAN.NAME == RESOURCE.NAME"

To:
TIMESPANTEST  = "((((TIMESPAN.FLOAT == 'N' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s) OR (TIMESPAN.FLOAT == 'Y' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s)) AND TIMESPAN.NAME == RESOURCE.NAME) OR 
(RESOURCE.RECURRANCE_MAX < %s))"

This will use the index for actually indexed events but also include 
every event beyond the index expansion date.
This will return "wrong" results and thus violate the CalDAV RFC section 
7.8 which says:

       "The response body for a successful CALDAV:calendar-query REPORT
       request MUST contain a DAV:response element for each iCalendar
       object that matched the search filter."

... but that would be ok IFF the only clients accessing the server do 
their own recurrence expansion and filtering.

>> The solution seems to be in the roadmap for DCS 1.3, however I would
>> regard this as a major flaw for 1.2 users.
> It certainly is.

Unfortunately, Apple have bumped the milestone for this bug to 2.x.
I don't understand that. This is a serious blocker. The calendar can not 
handle events like "same time next year" at all - which I would assume 
is often used.

/Peter